Multi-modal large model training method, document understanding method and equipment

Abstract

The invention provides a multi-modal large model training method, a document understanding method and equipment, and relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, natural language processing, computer vision, large models and the like. According to the specific implementation scheme, training data are obtained, and the training data comprise sample document images under a document understanding task, sample questions corresponding to the sample document images and sample answers of the sample questions; the sample document image and the sample question are input into a first multi-modal large model, a prediction answer is obtained, and the first multi-modal large model is a pre-trained multi-modal large model irrelevant to the document understanding task; and training the first multi-modal large model according to the prediction answer and the sample answer to obtain a second multi-modal large model. Therefore, the model training difficulty and the model training cost are reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as deep learning, natural language processing, computer vision, and large models, and in particular to a training method, document understanding method, and device for a multimodal large model. Background Art

[0002] With the continuous advancement of artificial intelligence technology, deep learning and natural language processing (NLP) have made significant progress, especially in document understanding tasks. However, document understanding tasks often involve more than just text processing. In many cases, they also involve document images and questions about the document images. Therefore, how to obtain a large multimodal model that can be used to perform document understanding tasks is an urgent technical problem. Summary of the invention

[0003] The present invention provides a training method for a multimodal large model, a document understanding method and a device.

[0004] According to one aspect of the present disclosure, a method for training a multimodal large model is provided, the method comprising: obtaining training data, wherein the training data comprises sample document images under a document understanding task, sample questions corresponding to the sample document images, and sample answers to the sample questions; inputting the sample document images and the sample questions into a first multimodal large model to obtain predicted answers, wherein the first multimodal large model is a pre-trained multimodal large model unrelated to the document understanding task; and training the first multimodal large model according to the predicted answers and the sample answers to obtain a second multimodal large model.

[0005] According to another aspect of the present disclosure, a document understanding method is provided, the method comprising: obtaining a document image to be processed in a document understanding task and a target question of the document image to be processed; obtaining a second multimodal large model; the second multimodal large model is obtained based on the training method of the multimodal large model as described above; inputting the document image to be processed and the target question into the second multimodal large model, and obtaining an answer output by the second multimodal large model.

[0006] According to another aspect of the present disclosure, a training device for a multimodal large model is provided, the device comprising: a first acquisition module, used to acquire training data, wherein the training data comprises sample document images under a document understanding task, sample questions corresponding to the sample document images, and sample answers to the sample questions; a second acquisition module, used to input the sample document images and the sample questions into the first multimodal large model to acquire predicted answers, wherein the first multimodal large model is a pre-trained multimodal large model unrelated to the document understanding task; a training module, used to train the first multimodal large model according to the predicted answers and the sample answers to obtain a second multimodal large model.

[0007] According to another aspect of the present disclosure, a document understanding device is provided, comprising: a third acquisition module, used to acquire a document image to be processed and a target question of the document image to be processed in a document understanding task; a fourth acquisition module, used to acquire a second multimodal large model; the second multimodal large model is obtained based on the training method of the multimodal large model as described above; and a fifth acquisition module, used to input the document image to be processed and the target question into the second multimodal large model to acquire an answer output by the second multimodal large model.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method of the multimodal large model proposed above in the present disclosure; or, execute the document understanding method proposed above in the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the multimodal large model training method proposed above in the present disclosure; or, execute the document understanding method proposed above in the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the training method of the multimodal large model proposed above in the present disclosure; or, implements the steps of the document understanding method proposed above in the present disclosure.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0014] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0015] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0016] Figure 4 It is a schematic diagram of the process of training the first multimodal large model in combination with the structure of the first multimodal large model;

[0017] Figure 5 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0018] Figure 6 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0019] Figure 7 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0020] Figure 8 is a schematic diagram according to a seventh embodiment of the present disclosure;

[0021] Fig. 9 is a schematic diagram according to an eighth embodiment of the present disclosure;

[0022] Fig.10 is a schematic diagram according to a ninth embodiment of the present disclosure;

[0023] Fig.11 It is a block diagram of an electronic device used to implement the multimodal large model training method or document understanding method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] At present, document understanding tasks often involve not only text processing, but also document images and questions about the document images, etc. Therefore, how to obtain a large multimodal model that can be used to perform document understanding tasks is an urgent technical problem.

[0026] In response to the above problems, the present disclosure proposes a training method for a multimodal large model, a document understanding method and a device.

[0027] Figure 1 It is a schematic diagram according to the first embodiment of the present disclosure. It should be noted that the training method of the multimodal large model of the embodiment of the present disclosure can be applied to the training device of the multimodal large model. The device can be an electronic device, or can be configured in an electronic device so that the electronic device can perform the training function of the multimodal large model.

[0028] Among them, the electronic device can be any device with computing capabilities, such as a personal computer (PC), a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, a smart speaker, a server, a server cluster, and other hardware devices with various operating systems, touch screens and / or display screens.

[0029] The training device for the multimodal large model may also be software in an electronic device, such as training software for the multimodal large model, etc. In the following embodiments, the training device for the multimodal large model is an electronic device as an example for description.

[0030] like Figure 1 As shown, the training method of the multimodal large model may include the following steps:

[0031] Step 101, obtaining training data, wherein the training data includes sample document images under a document understanding task, sample questions corresponding to the sample document images, and sample answers to the sample questions.

[0032] In an embodiment of the present disclosure, sample document images, sample questions corresponding to the sample document images, and sample answers to the sample questions may be obtained in an application scenario of a document understanding task.

[0033] The sample question refers to the question text raised in order to complete the document understanding task of the sample document image.

[0034] The sample answer refers to the answer text obtained for the sample question.

[0035] The application scenarios of document understanding tasks may include but are not limited to: financial reports, laws and regulations, education and research, instructions, contracts, research reports, etc.

[0036] In some exemplary embodiments, the sample document images include multiple types of document images, so that the second multimodal large model trained subsequently can accurately process various document images in the document understanding task, thereby improving the processing effect of the second multimodal large model on the document understanding task.

[0037] In some exemplary embodiments, the ratio between the data amounts of multiple types of document images in the training data is a preset ratio. Thus, the first multimodal large model can learn rich and diverse information from different types of data, which helps to improve the generalization ability of the first multimodal large model and reduce the risk of overfitting.

[0038] The preset ratio is preset according to actual needs. For example, the training data includes two types of document images, namely table document images and chart document images. The ratio between the data volume of table document images and chart document images in the training data can be 0.4:0.6, or 0.3:0.7, etc.

[0039] In some exemplary embodiments, the document images may be classified into table-type document images, text-type document images, and chart-type document images according to the differences in the main contents of the document images.

[0040] The table document image refers to an image obtained by scanning or photographing a document mainly containing table data.

[0041] The chart document image refers to an image obtained by scanning or photographing a document mainly containing chart data.

[0042] The text document image refers to an image obtained by scanning or photographing a document mainly including text information.

[0043] In some exemplary embodiments, the multiple types of document images include at least two of the following: table-type document images, text-type document images, and chart-type document images. Thus, the first multimodal large model can be trained in combination with the multiple types of document images, so that the trained second multimodal large model can process various types of document images in the document understanding task, thereby improving the processing effect of the second multimodal large model on the document understanding task.

[0044] Step 102: Input the sample document image and the sample question into the first multimodal large model to obtain a predicted answer, wherein the first multimodal large model is a pre-trained multimodal large model that is unrelated to the document understanding task.

[0045] In the disclosed embodiment, any multimodal large model used in a general question-answering scenario may be used as the first multimodal large model.

[0046] Step 103: train the first multimodal large model according to the predicted answers and the sample answers to obtain a second multimodal large model.

[0047] It can be understood that since the second multimodal large model is obtained by training the first multimodal large model based on the sample data under the document understanding task, the second multimodal large model can perform the document understanding task and perform well in processing the document understanding task while ensuring that the general understanding ability of the second multimodal large model will not be reduced too much.

[0048] The training method of the multimodal large model of the embodiment of the present disclosure obtains training data, wherein the training data includes sample document images under the document understanding task, sample questions corresponding to the sample document images, and sample answers to the sample questions; the sample document images and sample questions are input into the first multimodal large model to obtain predicted answers, wherein the first multimodal large model is a pre-trained multimodal large model that is unrelated to the document understanding task; according to the predicted answers and the sample answers, the first multimodal large model is trained to obtain a second multimodal large model. Thus, the first multimodal large model is trained in combination with the sample data under the document understanding task, and the trained second multimodal large model can perform the document understanding task, thereby reducing the difficulty and cost of model training.

[0049] In order to further reduce the model training cost, when the sample question of the sample document image is not answered, the sample answer of the sample question can be determined in combination with the text content in the sample document image to reduce the manual determination of the sample answer of the sample question. Figure 2 This situation is described as an example. Figure 2 As shown, Figure 2 is a schematic diagram according to a second embodiment of the present disclosure, Figure 2 The illustrated embodiment may include the following steps:

[0050] Step 201: determine sample document images and sample questions of the sample document images under the document understanding task.

[0051] In the disclosed embodiment, in order to have sufficient sample document images and corresponding sample questions, the process of the electronic device executing step 201 may be, for example: obtaining a real document image in the application scenario of the document understanding task, and obtaining candidate questions set for the real document image; generating multiple sample document images based on the real document image, wherein there is a difference in content between the sample document image and the real document image, and there is a difference in content between different sample document images; and using the candidate questions as sample questions of the sample document images. Thus, in combination with the real document images in the application scenario of the document understanding task, multiple sample document images are constructed, and sample questions of the sample document images are constructed based on the questions of the real document images, so that the constructed sample document images and sample questions are closer to the real data in the application scenario of the document understanding task, and then the accuracy of the trained second multimodal large model in processing the document understanding task can be improved.

[0052] In another embodiment of the present disclosure, the process of the electronic device executing step 201 may be, for example, obtaining a real document image in the application scenario of the document understanding task, determining the document image type of the real document image based on the main content contained in the real document image, and obtaining a question template corresponding to the document image type, constructing multiple sample document images based on the real document image, and constructing sample questions of the sample document images based on the question template. Thus, multiple sample document images are constructed in combination with the real document images in the application scenario of the document understanding task, and sample questions of the sample document images are constructed based on the real document images, so that the constructed sample document images and sample questions are closer to the real data in the application scenario of the document understanding task, and then the accuracy of the trained second multimodal large model in processing the document understanding task can be improved.

[0053] Step 202: determine the first text content in the sample document image.

[0054] In some exemplary embodiments, the language type corresponding to the first text content may be a first language. For example, the first language may be Chinese.

[0055] It should be noted that, in this embodiment, the language type of the text content in the document image is Chinese as an example for exemplary description.

[0056] It can be understood that in different application scenarios, the implementation methods of determining the first text content in the sample document image are different, and an exemplary description is as follows:

[0057] As an example, optical character recognition (OCR) may be performed on the sample document image to obtain the first text content.

[0058] In some exemplary embodiments, the sample document image may be processed by an OCR tool or an OCR model to obtain the text content in the sample document image. Thus, the first text content in the sample document image is accurately determined by optical character recognition.

[0059] As another example, a sample document image and a first preset question are input into the third multimodal large model, and the output first text content is obtained, wherein the first preset question is used to indicate the determination of the text content in the sample document image. Thus, the second text content in the sample document image is quickly and accurately determined by the third multimodal large model.

[0060] Among them, the third multimodal large model in this embodiment can be the same as the first multimodal large model, or the third multimodal large model can be a multimodal large model with a larger number of model parameters than the first multimodal large model. This embodiment limits the third multimodal large model.

[0061] The first preset question is used to indicate the determination of the text content in the sample document image. That is, the first preset question is a question used to determine the text content in the sample document image, for example, the first preset question may be: what words are included in the image, or what words are in the image.

[0062] Step 203: Determine a sample answer to the sample question based on the first text content and the sample question.

[0063] In one embodiment of the present disclosure, the first text content and the sample question can be input into a large language model of a single modality, so as to output a sample answer to the sample question through the large language model. Thus, the trouble of manually determining the answer to the sample question is reduced, thereby reducing the cost in the model training process.

[0064] Step 204: input the sample document image and the sample question into the first multimodal large model to obtain a predicted answer, wherein the first multimodal large model is a pre-trained multimodal large model that is unrelated to the document understanding task.

[0065] It can be understood that the sample document images may include various sizes. In order to enable the trained second multimodal large model to process images of different sizes, improve the performance of the second multimodal large model in practical applications, and avoid insufficient video memory, the sample document images can be pre-processed in combination with a preset size threshold.

[0066] In one embodiment of the present disclosure, when the image size of the sample document image does not exceed a preset size threshold, the sample document image and the sample question are input into the first multimodal large model to obtain a predicted answer.

[0067] The preset size threshold, which may also be referred to as a preset resize threshold, is an image size threshold that is preset according to actual needs.

[0068] In another embodiment of the present disclosure, when the image size of the sample document image exceeds a preset size threshold, the sample document image is scaled to obtain a scaled sample document image, wherein the image size of the scaled sample document image is less than or equal to the preset size threshold. Correspondingly, inputting the sample document image and the sample question into the first multimodal large model to obtain a predicted answer can be replaced by: inputting the scaled sample document image and the sample question into the first multimodal large model to obtain a predicted answer.

[0069] Based on the above description, it can be seen that in this embodiment, when it is determined that the image size of the sample document image exceeds the preset size threshold, the sample document image is scaled, and when the image size of the sample document image does not exceed the preset size threshold, the sample document image is not processed. Therefore, in combination with the preset size threshold, the sample document image is preprocessed, so that the training data includes sample document images of various image sizes, and the first multimodal large model is trained by sample document images of various image sizes, so that the trained output second multimodal large model can adapt to inputs of different sizes, thereby improving the performance of the second multimodal large model in practical applications.

[0070] Step 205: Train the first multimodal large model according to the predicted answers and the sample answers to obtain a second multimodal large model.

[0071] In one embodiment of the present disclosure, the process of the electronic device executing step 205 may, for example, be: determining the loss function value of the first multimodal large model based on the predicted answer and the sample answer; and performing parameter adjustment processing on the first multimodal large model based on the loss function value to obtain the second multimodal large model.

[0072] In an embodiment of the present disclosure, the electronic device may determine the loss function value of the first multimodal large model by combining the sample answer, the predicted answer, and the loss function of the first multimodal large model.

[0073] In this embodiment, the loss function value of the first multimodal large model is determined by combining the predicted answer and the sample answer, and then the parameters of the first multimodal large model are adjusted based on the loss function value, which can improve the training speed and accuracy of the model.

[0074] The training method of the multimodal large model of the embodiment of the present disclosure determines sample document images and sample questions of the sample document images under document understanding, and can automatically determine sample answers to the sample questions based on the first text content in the sample document images and the sample questions, thereby reducing the trouble of manually determining the sample questions of the sample questions, and inputs the sample document images and the sample questions into the first multimodal large model to obtain predicted answers, and trains the first multimodal large model based on the predicted answers and the sample answers to obtain a second multimodal large model, which can further improve the training speed of the model and reduce the training cost of the model.

[0075] Wherein, in the case where the first multimodal large model includes: a first visual encoder, a first cross-modal connector, and a first large language model, in order to clearly explain the process of the first multimodal large model determining the predicted answer, the following is combined with Figure 3 This process is described exemplarily.

[0076] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure.

[0077] like Figure 3 As shown, the method may include the following steps:

[0078] Step 301 : obtaining training data, wherein the training data includes sample document images under a document understanding task, sample questions corresponding to the sample document images, and sample answers to the sample questions.

[0079] It should be noted that, for the specific description of step 301, reference may be made to the relevant descriptions in other embodiments, which will not be repeated here.

[0080] Step 302: input the sample document image into a first visual encoder to obtain a first visual encoding result of the sample document image.

[0081] In this embodiment, after the sample document image is input into the first visual encoder, the first visual encoder divides the sample document image into multiple visual units and performs encoding processing on each visual unit to obtain a first visual encoding result of the sample document image.

[0082] Step 303: input the first visual encoding result into the first cross-modal connector, and map the first visual encoding result from the visual space to the text space through the first cross-modal connector to obtain a first text feature of the sample document image.

[0083] Step 304: Input the sample question and the first text feature into the first language model to obtain an output predicted answer.

[0084] In this embodiment, the sample document image is processed through the visual encoder, which can better retain and extract the features of the sample document image, and the features in the visual space are mapped to the text space through the first cross-modal connector, so that the features of the sample document image and the sample question are aligned in the text space, which facilitates the subsequent first language model to accurately understand the semantics and obtain the predicted answer.

[0085] Step 305: Train the first large language model in the first multimodal large model according to the predicted answer and the sample answer to obtain a second multimodal large model.

[0086] In one embodiment of the present disclosure, the process of the electronic device executing step 305 may, for example, be: determining the loss function value of the first large language model based on the predicted answer and the sample answer, and fine-tuning the parameters of the first large language model based on the loss function value to obtain the second multimodal large model.

[0087] The training method of the multimodal large model of the embodiment of the present disclosure inputs a sample document image into a first visual encoder to obtain a first visual encoding result of the sample document image, and inputs the first visual encoding result into a first cross-modal connector to map the first visual encoding result from a visual space to a text space through the first cross-modal connector to obtain a first text feature of the sample document image, and inputs a sample question and the first text feature into a first large language model to obtain an output predicted answer, and trains the first large language model in the first multimodal large model based on the predicted answer and the sample answer to obtain a second multimodal large model, which can further improve the training speed of the model and reduce the training cost of the model.

[0088] The following example illustrates this. Figure 4 It is a schematic diagram of the process of training the first multimodal large model in combination with the structure of the first multimodal large model. Figure 4 The following steps may be included: Step 401, when it is determined that the image size of the sample document image does not exceed a preset size threshold, the sample document image is input into a first visual encoder. Step 402, the visual encoding result of the first visual encoder is input into a first cross-modal connector, and the first cross-modal connector maps the visual encoding result from the visual space to the text space to obtain a first text feature in the text space. Step 403, the sample question and the first text feature are input into a first large language model to obtain a predicted answer. Step 404, the first large language model is fine-tuned in combination with the predicted answer and the sample answer.

[0089] Figure 5This is a schematic diagram according to the fourth embodiment of the present disclosure. It should be noted that the document understanding method of the embodiment of the present disclosure can be applied to a document understanding device, which can be an electronic device, or can be configured in an electronic device so that the electronic device can perform a document processing function. In the following embodiments, the execution subject is an electronic device as an example for explanation.

[0090] Among them, the electronic device can be any device with computing capabilities, such as a personal computer (PC), a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, a smart speaker, a server, a server cluster, and other hardware devices with various operating systems, touch screens and / or display screens.

[0091] The document understanding device may also be software in an electronic device, such as document understanding software.

[0092] like Figure 5 As shown, the document understanding method may include the following steps:

[0093] Step 501: Obtain a document image to be processed and a target question of the document image to be processed in a document understanding task.

[0094] The target question refers to the question text raised in order to complete the document understanding task of the document image to be processed.

[0095] For example, the document image to be processed is an image obtained by photographing or scanning a project contract document. Correspondingly, the target question may be “What is the name of the project?”.

[0096] For another example, the document image to be processed is an image obtained by photographing or scanning the functional specification document. Correspondingly, the target question may be: "What is the content of the switch key in the functional specification?".

[0097] Step 502, obtaining a second multimodal large model; the second multimodal large model is obtained based on the training method of the multimodal large model disclosed in any one of the aforementioned embodiments.

[0098] Step 503: input the document image to be processed and the target question into the second multimodal large model to obtain the answer output by the second multimodal large model.

[0099] In one embodiment of the present disclosure, when the second multimodal large model includes: a second visual encoder, a second cross-modal connector, and a second large language model, the process of the electronic device executing step 503 may be, for example: inputting the document image to be processed into the second visual encoder to obtain the second visual encoding result of the document image to be processed; inputting the second visual encoding result into the second cross-modal connector, mapping the second visual encoding result from the visual space to the text space through the second cross-modal connector, and obtaining the second text feature of the document image to be processed; inputting the target question and the second text feature into the second large language model to obtain the output answer. Thus, by processing the document image to be processed through the second visual encoder, the features of the document image to be processed can be better retained and extracted, and the features in the visual space are mapped to the text space through the second cross-modal connector, so that the features of the document image to be processed and the target question are aligned in the text space, which facilitates the subsequent first large language model to accurately understand the semantics and accurately obtain the answer to the target question.

[0100] The document understanding method of the disclosed embodiment obtains the document image to be processed and the target question of the document image to be processed in the document understanding task, obtains the second multimodal large model obtained by the training method of the multimodal large model disclosed in any of the aforementioned embodiments, and inputs the document image to be processed and the target question into the second multimodal large model to obtain the answer output by the second multimodal large model. Thus, the second multimodal large model can accurately process the document processing task and accurately obtain the answer to the target question in the document processing task.

[0101] Figure 6 is a schematic diagram according to a fifth embodiment of the present disclosure.

[0102] like Figure 6 As shown, the method may include:

[0103] Step 601: Obtain a document image to be processed and a target question of the document image to be processed in a document understanding task.

[0104] Step 602, obtaining a second multimodal large model; the second multimodal large model is obtained based on the training method of the multimodal large model disclosed in any one of the aforementioned embodiments.

[0105] It should be noted that, for the specific description of step 601 and step 602, reference may be made to the relevant description in other embodiments, which will not be repeated here.

[0106] Step 603: proportionally enlarge the document image to be processed by a preset multiple to obtain an enlarged document image to be processed.

[0107] In this embodiment, in order to enable the second multimodal large model to obtain more comprehensive visual features so as to improve the understanding of the second multimodal large model, the document image to be processed can be proportionally enlarged by a preset multiple before being input into the second multimodal large model.

[0108] The preset multiple is preset according to actual needs. For example, the preset multiple may be 1.1, 1.2 or 1.3, etc. This embodiment does not specifically limit the value of the preset multiple.

[0109] In some examples, the preset multiple ranges from 1.1 to 1.3.

[0110] Step 604: input the enlarged document image to be processed and the target question into the second multimodal large model to obtain the answer output by the second multimodal large model.

[0111] In one embodiment of the present disclosure, when the second multimodal large model includes: a second visual encoder, a second cross-modal connector and a second large language model, the process of the electronic device executing step 603 may, for example, be: inputting the enlarged document image to be processed into the second visual encoder to obtain a second visual encoding result of the enlarged document image to be processed; inputting the second visual encoding result into the second cross-modal connector, mapping the second visual encoding result from the visual space to the text space through the second cross-modal connector to obtain a second text feature; inputting the target question and the second text feature into the second large language model to obtain an output answer.

[0112] The document understanding method of the disclosed embodiment obtains the document image to be processed and the target question of the document image to be processed in the document understanding task, obtains the second multimodal large model obtained by the training method of the multimodal large model disclosed in any of the aforementioned embodiments, and proportionally enlarges the document image to be processed by a preset multiple, and inputs the enlarged document image to be processed and the target question into the second multimodal large model to obtain the answer output by the second multimodal large model. Thus, the second multimodal large model can obtain more comprehensive visual features in the image to be processed, which helps to improve the accuracy of the answer generated by the second multimodal large model.

[0113] Figure 7 is a schematic diagram according to a sixth embodiment of the present disclosure.

[0114] Step 701: Obtain a document image to be processed and a target question of the document image to be processed in a document understanding task.

[0115] It should be noted that, for the specific description of step 701, reference may be made to the relevant descriptions in other embodiments, which will not be repeated here.

[0116] Step 702, obtaining a second multimodal large model; the second multimodal large model is obtained based on the training method of the multimodal large model disclosed in any one of the aforementioned embodiments.

[0117] It should be noted that for the specific description of step 701 and step 702, reference may be made to the relevant description in other embodiments, which will not be repeated here.

[0118] Step 703: Determine the second text content in the document image to be processed.

[0119] In this embodiment, optical character recognition (OCR) can be performed on the document image to be processed to obtain the second text content, thereby accurately determining the second text content of the document image to be processed.

[0120] In some exemplary embodiments, optical character recognition (OCR) may be performed on the document image to be processed by an OCR tool or an OCR model to obtain the second text content.

[0121] Step 704: input the document image to be processed, the target question and the second text content into the second multimodal large model to obtain the answer output by the second multimodal large model.

[0122] In some exemplary embodiments, when the second multimodal large model includes: a second visual encoder, a second cross-modal connector, and a second large language model, the process of the electronic device performing step 703 may be, for example: splicing the target question and the second text content to obtain a spliced ​​text, inputting the document image to be processed into the second visual encoder to obtain a second visual encoding result of the document image to be processed; inputting the second visual encoding result into the second cross-modal connector, mapping the second visual encoding result from the visual space to the text space through the second cross-modal connector to obtain a second text feature; inputting the spliced ​​text and the second text feature into the second large language model to obtain an output answer. In this way, the accuracy of the answer generated by the second large language model can be further improved.

[0123] The document understanding method of the embodiment of the present disclosure obtains the document image to be processed and the target question of the document image to be processed in the document understanding task, obtains the second multimodal large model obtained by the training method of the multimodal large model disclosed in any of the aforementioned embodiments, determines the second text content in the document image to be processed, and inputs the document image to be processed, the target question and the second text content into the second multimodal large model to obtain the answer output by the second multimodal large model. In this way, the accuracy of the answer generated by the second multimodal large model can be further improved.

[0124] Figure 8 is a schematic diagram according to a seventh embodiment of the present disclosure.

[0125] Step 801, obtaining a document image to be processed and a target question of the document image to be processed in a document understanding task.

[0126] In some exemplary embodiments, in order to enable the second multimodal large model to obtain more comprehensive visual features so as to improve the understanding of the second multimodal large model, the document image to be processed may also be proportionally magnified by a preset multiple.

[0127] Step 802, obtaining a second multimodal large model; the second multimodal large model is obtained based on the training method of the multimodal large model disclosed in any one of the aforementioned embodiments.

[0128] It should be noted that, for the specific description of step 801 and step 802, reference may be made to the relevant description in other embodiments, which will not be repeated here.

[0129] Step 803: input the document image to be processed and the second preset question into the second multimodal large model to obtain the output second text content, wherein the second preset question is used to indicate the determination of the text content in the document image to be processed.

[0130] Among them, the second preset question is used to indicate the determination of the text content in the document image to be processed, that is, the second preset question is a question used to determine the text content in the sample document image. For example, the second preset question may be: what text is included in the document image to be processed, or what text is in the image, etc.

[0131] Step 804: input the document image to be processed, the target question and the second text content into the second multimodal large model to obtain the answer output by the second multimodal large model.

[0132] In some exemplary embodiments, when the second multimodal large model includes: a second visual encoder, a second cross-modal connector, and a second large language model, the process of the electronic device performing step 803 may be, for example: splicing the target question and the second text content to obtain a spliced ​​text, inputting the document image to be processed into the second visual encoder to obtain a second visual encoding result of the document image to be processed; inputting the second visual encoding result into the second cross-modal connector, mapping the second visual encoding result from the visual space to the text space through the second cross-modal connector, and obtaining a second text feature of the document image to be processed; inputting the spliced ​​text and the second text feature into the second large language model to obtain an output answer. In this way, the accuracy of the answer generated by the second large language model can be further improved.

[0133] The document understanding method of the disclosed embodiment obtains the document image to be processed and the target question of the document image to be processed in the document understanding task, and obtains the second multimodal large model obtained by the training method of the multimodal large model disclosed in any of the aforementioned embodiments, and inputs the document image to be processed and the second preset question into the second multimodal large model to obtain the output second text content, and inputs the document image to be processed, the target question and the second text content into the second multimodal large model to obtain the answer output by the second multimodal large model. Thus, the second multimodal large model first determines the second text content in the document image to be processed by means of a Chain of Thought (CoT), and further determines the answer to the target question in combination with the second text content, the document image to be processed and the target question, which can further improve the accuracy of the answer generated by the second multimodal large model.

[0134] In order to implement the above embodiments, the present disclosure also provides a training device for a multimodal large model.

[0135] Fig. 9 is a schematic diagram according to an eighth embodiment of the present disclosure.

[0136] like Fig. 9 As shown, the multimodal large model training device 90 may include: a first acquisition module 901, a second acquisition module 902 and a training module 903.

[0137] The first acquisition module 901 is used to acquire training data, wherein the training data includes sample document images under the document understanding task, sample questions corresponding to the sample document images, and sample answers to the sample questions.

[0138] The second acquisition module 902 is used to input the sample document image and the sample question into the first multimodal large model to obtain the predicted answer, wherein the first multimodal large model is a pre-trained multimodal large model that is unrelated to the document understanding task.

[0139] The training module 903 is used to train the first multimodal large model according to the predicted answers and the sample answers to obtain the second multimodal large model.

[0140] As a possible implementation of the embodiment of the present disclosure, the first acquisition module 901 includes:

[0141] A first determining unit, used to determine a sample document image and a sample question of the sample document image under the document understanding task;

[0142] A second determining unit, configured to determine a first text content in the sample document image;

[0143] The third determining unit is used to determine a sample answer to the sample question according to the first text content and the sample question.

[0144] As a possible implementation method of the embodiment of the present disclosure, the second determination unit is specifically used to: perform optical character recognition (OCR) on the sample document image to obtain the first text content; or, input the sample document image and the first preset question into the third multimodal large model to obtain the output first text content, wherein the first preset question is used to indicate the determination of the text content in the sample document image.

[0145] As a possible implementation method of an embodiment of the present disclosure, the first determination unit is specifically used to: obtain a real document image in an application scenario of a document understanding task, and obtain candidate questions set for the real document image; generate multiple sample document images based on the real document image, wherein there are differences in content between the sample document image and the real document image, and there are differences in content between different sample document images; and use the candidate questions as sample questions of the sample document image.

[0146] As a possible implementation method of an embodiment of the present disclosure, the first multimodal large model includes: a first visual encoder, a first cross-modal connector and a first large language model, and the second acquisition module 902 is specifically used to: input the sample document image into the first visual encoder to obtain a first visual encoding result of the sample document image; input the first visual encoding result into the first cross-modal connector, map the first visual encoding result from the visual space to the text space through the first cross-modal connector, and obtain a first text feature of the sample document image; input the sample question and the first text feature into the first large language model to obtain the output predicted answer.

[0147] As a possible implementation method of the embodiment of the present disclosure, the training module 903 is specifically used to: determine the loss function value of the first multimodal large model based on the predicted answer and the sample answer; and adjust the parameters of the first multimodal large model based on the loss function value to obtain the second multimodal large model.

[0148] As a possible implementation method of the embodiment of the present disclosure, the first acquisition module 901 is specifically used to: when the image size of the sample document image does not exceed a preset size threshold, input the sample document image and the sample question into the first multimodal large model to obtain a predicted answer.

[0149] As a possible implementation method of the embodiment of the present disclosure, the first acquisition module 901 is also used to: when the image size of the sample document image exceeds a preset size threshold, scale the sample document image to obtain a scaled sample document image, wherein the image size of the scaled sample document image is less than or equal to the preset size threshold; input the scaled sample document image and the sample question into the first multimodal large model to obtain a predicted answer.

[0150] As a possible implementation of the embodiment of the present disclosure, the sample document image includes multiple types of document images.

[0151] As a possible implementation manner of the embodiment of the present disclosure, the ratio between the data amounts of multiple types of document images in the training data is a preset ratio.

[0152] As a possible implementation manner of the embodiment of the present disclosure, the multiple types of document images include at least two of the following: table-type document images, text-type document images, and chart-type document images.

[0153] The training device of the multimodal large model of the embodiment of the present disclosure obtains training data, wherein the training data includes sample document images under the document understanding task, sample questions corresponding to the sample document images, and sample answers to the sample questions; the sample document images and sample questions are input into the first multimodal large model to obtain predicted answers, wherein the first multimodal large model is a pre-trained multimodal large model unrelated to the document understanding task; according to the predicted answers and the sample answers, the first multimodal large model is trained to obtain a second multimodal large model. Thus, the first multimodal large model is trained in combination with the sample data under the document understanding task, and the trained second multimodal large model can perform the document understanding task, thereby reducing the difficulty and cost of model training.

[0154] In order to implement the above embodiments, the present disclosure also provides a document understanding device.

[0155] Fig.10 is a schematic diagram of a ninth embodiment of the present disclosure.

[0156] like Fig.10 As shown, the document understanding device 100 may include: a third acquisition module 1001 , a fourth acquisition module 1002 and a fifth acquisition module 1003 .

[0157] The third acquisition module 1001 is used to acquire the document image to be processed and the target question of the document image to be processed in the document understanding task.

[0158] The fourth acquisition module 1002 is used to acquire a second multimodal large model; the second multimodal large model is obtained based on the training method of the multimodal large model disclosed in the embodiment of the present disclosure.

[0159] The fifth acquisition module 1003 is used to input the document image to be processed and the target question into the second multimodal large model to obtain the answer output by the second multimodal large model.

[0160] As a possible implementation of the embodiment of the present disclosure, the device further includes:

[0161] The image magnification module is used to proportionally magnify the document image to be processed by a preset multiple.

[0162] As a possible implementation method of an embodiment of the present disclosure, the second multimodal large model includes: a second visual encoder, a second cross-modal connector and a second large language model, and a fourth acquisition module 1002, which is specifically used to: input the document image to be processed into the second visual encoder to obtain a second visual encoding result of the document image to be processed; input the second visual encoding result into the second cross-modal connector, and map the second visual encoding result from the visual space to the text space through the second cross-modal connector to obtain a second text feature of the document image to be processed; input the target question and the second text feature into the second large language model to obtain the output answer.

[0163] As a possible implementation of the embodiment of the present disclosure, the device further includes:

[0164] A sixth determination module, used to determine the second text content in the document image to be processed;

[0165] Among them, the fourth acquisition module 1002 is specifically used to: input the document image to be processed, the target question and the second text content into the second multimodal large model, and obtain the answer output by the second multimodal large model.

[0166] As a possible implementation method of the embodiment of the present disclosure, the sixth determination module is specifically used to: perform optical character recognition (OCR) on the document image to be processed to obtain the second text content; or, input the document image to be processed and the second preset question into the second multimodal large model to obtain the output second text content, wherein the second preset question is used to indicate the determination of the text content in the document image to be processed.

[0167] The document understanding device of the disclosed embodiment obtains the document image to be processed and the target question of the document image to be processed in the document understanding task, obtains the second multimodal large model obtained by the training method of the multimodal large model disclosed in any of the aforementioned embodiments, and inputs the document image to be processed and the target question into the second multimodal large model to obtain the answer output by the second multimodal large model. Thus, the second multimodal large model can accurately process the document processing task and accurately obtain the answer to the target question in the document processing task.

[0168] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's consent, comply with the relevant laws and regulations, and do not violate public order and good morals.

[0169] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0170] Fig.11 It is a block diagram of an electronic device for implementing a training method or a document understanding method of a multimodal large model of an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0171] like Fig.11 As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the electronic device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0172] Multiple components in the electronic device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the electronic device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0173] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as a training method for a multimodal large model or a document understanding method. For example, in some embodiments, the training method for a multimodal large model or a document understanding method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1100 via ROM 1102 and / or the communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the training method for a multimodal large model or a document understanding method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute a multimodal large model training method or a document understanding method in any other appropriate manner (e.g., by means of firmware).

[0174] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0175] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0176] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0177] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0178] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0179] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0180] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0181] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a multimodal large model, comprising: Acquire training data, wherein the training data includes sample document images under a document understanding task, sample questions corresponding to the sample document images, and sample answers to the sample questions; Inputting the sample document image and the sample question into a first multimodal large model to obtain a predicted answer, wherein the first multimodal large model is a pre-trained multimodal large model unrelated to the document understanding task; The first multimodal large model is trained according to the predicted answer and the sample answer to obtain a second multimodal large model.

2. The method according to claim 1, wherein: The obtaining of training data comprises: Determining sample document images under the document understanding task and sample questions of the sample document images; Determining first text content in the sample document image; A sample answer to the sample question is determined based on the first text content and the sample question.

3. The method according to claim 2, wherein: The determining of the first text content in the sample document image includes: Performing optical character recognition (OCR) on the sample document image to obtain the first text content; or, The sample document image and a first preset question are input into a third multimodal large model to obtain the first text content as output, wherein the first preset question is used to indicate the determination of the text content in the sample document image.

4. The method according to claim 2, wherein: The determining of the sample document image under the document understanding and the sample question of the sample document image comprises: Acquire a real document image in an application scenario of the document understanding task, and acquire candidate questions set for the real document image; Generating a plurality of sample document images according to the real document image, wherein the contents of the sample document image and the real document image are different, and the contents of different sample document images are different; The candidate questions are used as sample questions of the sample document image.

5. The method according to claim 1, wherein: The first multimodal large model includes: a first visual encoder, a first cross-modal connector and a first large language model; the step of inputting the sample document image and the sample question into the first multimodal large model to obtain a predicted answer includes: Inputting the sample document image into the first visual encoder to obtain a first visual encoding result of the sample document image; Inputting the first visual encoding result into the first cross-modal connector, mapping the first visual encoding result from a visual space to a text space through the first cross-modal connector, and obtaining a first text feature of the sample document image; The sample question and the first text feature are input into the first language model to obtain the output predicted answer.

6. The method according to claim 1, wherein: The step of training the first multimodal large model according to the predicted answer and the sample answer to obtain a second multimodal large model includes: Determining a loss function value of the first multimodal large model according to the predicted answer and the sample answer; According to the loss function value, parameter adjustment processing is performed on the first multimodal large model to obtain the second multimodal large model.

7. The method according to claim 1, wherein: The step of inputting the sample document image and the sample question into a first multimodal large model to obtain a predicted answer includes: When the image size of the sample document image does not exceed a preset size threshold, the sample document image and the sample question are input into the first multimodal large model to obtain a predicted answer.

8. The method according to claim 7, wherein: The step of inputting the sample document image and the sample question into the first multimodal large model to obtain a predicted answer further includes: In the case where the image size of the sample document image exceeds a preset size threshold, scaling the sample document image to obtain a scaled sample document image, wherein the image size of the scaled sample document image is less than or equal to the preset size threshold; The scaled sample document image and the sample question are input into the first multimodal large model to obtain the predicted answer.

9. The method according to any one of claims 1 to 8, wherein: The sample document images include multiple types of document images.

10. The method according to claim 9, wherein: The multiple types of document images include at least two of the following: table-type document images, text-type document images, and chart-type document images.

11. The method according to claim 9, wherein: The ratio between the data amounts of multiple types of document images in the training data is a preset ratio.

12. A document understanding method, the method comprising: Obtaining a document image to be processed in a document understanding task and a target problem of the document image to be processed; obtaining a second multimodal large model; The second multimodal large model is obtained based on the training method of the multimodal large model according to any one of claims 1 to 11; The document image to be processed and the target question are input into the second multimodal large model to obtain an answer output by the second multimodal large model.

13. The method according to claim 12, wherein: Before inputting the document image to be processed and the target question into the second multimodal large model and obtaining the answer output by the second multimodal large model, the method further includes: The document image to be processed is proportionally enlarged by a preset multiple.

14. The method according to claim 12, wherein: The second multimodal large model includes: a second visual encoder, a second cross-modal connector, and a second large language model; the step of inputting the document image to be processed and the target question into the second multimodal large model to obtain an answer output by the second multimodal large model includes: Inputting the document image to be processed into the second visual encoder to obtain a second visual encoding result of the document image to be processed; Inputting the second visual encoding result into the second cross-modal connector, mapping the second visual encoding result from a visual space to a text space through the second cross-modal connector, and obtaining a second text feature of the document image to be processed; The target question and the second text feature are input into the second language model to obtain the output answer.

15. The method according to claim 12, wherein: Before inputting the document image to be processed and the target question into the second multimodal large model and obtaining the answer output by the second multimodal large model, the method further includes: Determining second text content in the document image to be processed; The step of inputting the document image to be processed and the target question into the second multimodal large model and obtaining an answer output by the second multimodal large model includes: The document image to be processed, the target question and the second text content are input into the second multimodal large model to obtain an answer output by the second multimodal large model.

16. The method according to claim 15, wherein: The determining of the second text content in the document image to be processed includes: Performing optical character recognition (OCR) on the document image to be processed to obtain the second text content; or, The document image to be processed and a second preset question are input into the second multimodal large model to obtain the second text content as output, wherein the second preset question is used to indicate the determination of the text content in the document image to be processed.

17. A training device for a multimodal large model, comprising: A first acquisition module is used to acquire training data, wherein the training data includes sample document images under a document understanding task, sample questions corresponding to the sample document images, and sample answers to the sample questions; A second acquisition module is used to input the sample document image and the sample question into a first multimodal large model to obtain a predicted answer, wherein the first multimodal large model is a pre-trained multimodal large model that is unrelated to the document understanding task; A training module is used to train the first multimodal large model according to the predicted answer and the sample answer to obtain a second multimodal large model.

18. The device according to claim 17, wherein: The first acquisition module includes: A first determining unit, configured to determine a sample document image under the document understanding task and a sample question of the sample document image; A second determining unit, configured to determine a first text content in the sample document image; The third determining unit is used to determine a sample answer to the sample question according to the first text content and the sample question.

19. The device according to claim 18, wherein: The second determining unit is specifically configured to: Performing optical character recognition (OCR) on the sample document image to obtain the first text content; or, The sample document image and a first preset question are input into a third multimodal large model to obtain the first text content as output, wherein the first preset question is used to indicate the determination of the text content in the sample document image.

20. The device according to claim 18, wherein The first determining unit is specifically configured to: Acquire a real document image in an application scenario of the document understanding task, and acquire candidate questions set for the real document image; Generating a plurality of sample document images according to the real document image, wherein the contents of the sample document image and the real document image are different, and the contents of different sample document images are different; The candidate questions are used as sample questions of the sample document image.

21. The device according to claim 17, wherein: The first multimodal large model includes: a first visual encoder, a first cross-modal connector and a first large language model; the second acquisition module is specifically used to: Inputting the sample document image into the first visual encoder to obtain a first visual encoding result of the sample document image; Inputting the first visual encoding result into the first cross-modal connector, mapping the first visual encoding result from a visual space to a text space through the first cross-modal connector, and obtaining a first text feature of the sample document image; The sample question and the first text feature are input into the first language model to obtain the output predicted answer.

22. The device according to claim 17, wherein: The training module is specifically used for: Determining a loss function value of the first multimodal large model according to the predicted answer and the sample answer; According to the loss function value, parameter adjustment processing is performed on the first multimodal large model to obtain the second multimodal large model.

23. The device according to claim 17, wherein: The first acquisition module is specifically used to: When the image size of the sample document image does not exceed a preset size threshold, the sample document image and the sample question are input into the first multimodal large model to obtain a predicted answer.

24. The device according to claim 23, wherein: The first acquisition module is further used for: In the case where the image size of the sample document image exceeds a preset size threshold, scaling the sample document image to obtain a scaled sample document image, wherein the image size of the scaled sample document image is less than or equal to the preset size threshold; The scaled sample document image and the sample question are input into the first multimodal large model to obtain the predicted answer.

25. The device according to any one of claims 17 to 24, wherein: The sample document images include multiple types of document images.

26. The device according to claim 25, wherein The multiple types of document images include at least two of the following: table-type document images, text-type document images, and chart-type document images.

27. The device according to claim 25, wherein: The ratio between the data amounts of multiple types of document images in the training data is a preset ratio.

28. A document understanding device, the device comprising: A third acquisition module is used to acquire a document image to be processed and a target question of the document image to be processed in a document understanding task; A fourth acquisition module, used for acquiring a second multimodal large model; the second multimodal large model is obtained based on the training method of the multimodal large model according to any one of claims 1 to 11; The fifth acquisition module is used to input the document image to be processed and the target question into the second multimodal large model to obtain the answer output by the second multimodal large model.

29. The device according to claim 28, wherein The device also includes: The image magnification module is used to proportionally magnify the document image to be processed by a preset multiple.

30. The device according to claim 28, wherein The second multimodal large model includes: a second visual encoder, a second cross-modal connector and a second large language model; the fourth acquisition module is specifically used to: Inputting the document image to be processed into the second visual encoder to obtain a second visual encoding result of the document image to be processed; Inputting the second visual encoding result into the second cross-modal connector, mapping the second visual encoding result from a visual space to a text space through the second cross-modal connector, and obtaining a second text feature of the document image to be processed; The target question and the second text feature are input into the second language model to obtain the output answer.

31. The apparatus of claim 28, wherein: The device also includes: A sixth determining module, used for determining the second text content in the document image to be processed; Wherein, the fourth acquisition module is specifically used for: The document image to be processed, the target question and the second text content are input into the second multimodal large model to obtain an answer output by the second multimodal large model.

32. The device according to claim 31, wherein The sixth determination module is specifically used for: Performing optical character recognition (OCR) on the document image to be processed to obtain the second text content; or, The document image to be processed and a second preset question are input into the second multimodal large model to obtain the second text content as output, wherein the second preset question is used to indicate the determination of the text content in the document image to be processed.

33. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1 to 11; or, execute the method of any one of claims 12 to 16.

34. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 11; or to execute the method according to any one of claims 12 to 16.

35. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11; or, implements the method according to any one of claims 12 to 16.

Images