Method and apparatus for training a document reading model based on cross-modal information
The method trains a document reading model using cross-modal information and OCR to enhance document understanding, eliminating the need for source-specific extraction schemes and reducing labor costs.
Patent Information
- Application Number
- JP2022176330
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-12-27
- Filing Date
- 2022-11-02
- Publication Date
- 2024-12-04
- Estimated Expiration
- 2042-11-02
AI Technical Summary
Existing plain text-based reading comprehension techniques require labor-intensive and time-consuming information extraction schemes that vary depending on the document source.
A method for training a document reading model using cross-modal information, including optical character recognition (OCR) to obtain text and layout information from rich text documents, and inputting this information into a reading comprehension model to predict answers, eliminating the need for source-specific extraction schemes.
Improves the ability to understand rich text documents by utilizing cross-modal information, making the process more end-to-end and reducing labor costs.
Smart Images

Figure 0007594571000001 
Figure 0007594571000002 
Figure 0007594571000003
Abstract
Description
[Technical field]
[0001] The present application relates to the field of data processing, in particular to natural language processing (NLP) and deep learning technology, and in particular to a document reading comprehension model training method based on cross-modal information and a visual question answering method, device, electronic device, and storage medium for document. [Background technology]
[0002] In the related art, the plain text based reading comprehension technology needs to perform information extraction on the document text in advance to obtain text information from the document, but it is very time-consuming and labor-intensive because different extraction schemes need to be designed depending on the source of the document. Summary of the Invention
[0003] The present application provides a method, an apparatus, an electronic device, and a storage medium for training a document reading comprehension model based on cross-modal information.
[0004] According to a first aspect of the present application, there is provided a method for training a document reading comprehension model based on cross-modal information, comprising the steps of: obtaining a question sample and a rich text document sample, wherein the rich text document sample includes an actual answer to the question sample; performing optical character recognition (OCR) processing on image information of the rich text document sample to obtain text information and layout information of the rich text document sample; inputting the text information, the layout information and the image information of the rich text document sample into a pre-established reading comprehension model to obtain a predicted answer to the question sample; and training the reading comprehension model based on the actual answer and the predicted answer.
[0005] According to a second aspect of the present application, a visual question answering method for documents is provided, comprising: a step of obtaining a candidate rich text document corresponding to received question information in response to the question information; a step of performing optical character recognition (OCR) processing on image information of the candidate rich text document to obtain text information and layout information of the candidate rich text document; a step of dividing the candidate rich text document at a predetermined step size to obtain a plurality of paragraph texts, and a step of splicing the question information with each of the paragraph texts respectively to obtain a plurality of paragraph splice texts; and a step of generating answer information corresponding to the question information based on the plurality of paragraph splice texts, the layout information and the image information.
[0006] According to a third aspect of the present application, there is provided a document reading comprehension model training device based on cross-modal information, comprising: an acquisition module for acquiring a question sample and a rich text document sample, where the rich text document sample includes an actual answer to the question sample; a first processing module for performing optical character recognition (OCR) processing on image information of the rich text document sample to acquire text information and layout information of the rich text document sample; a second processing module for inputting the text information, the layout information and the image information of the rich text document sample into a predetermined reading comprehension model to acquire a predicted answer to the question sample; and a training module for training the reading comprehension model based on the actual answer and the predicted answer.
[0007] According to a fourth aspect of the present application, a visual question answering device for documents is provided, comprising: an acquisition module for acquiring a candidate rich text document corresponding to received question information in response to the question information; a first processing module for performing optical character recognition (OCR) processing on image information of the candidate rich text document to acquire text information and layout information of the candidate rich text document; a second processing module for dividing the candidate rich text document at a predetermined step size to acquire a plurality of paragraph texts, and splicing the question information with each of the paragraph texts to acquire a plurality of paragraph splice texts; and a generation module for generating answer information corresponding to the question information based on the plurality of paragraph splice texts, the layout information and the image information.
[0008] According to a fifth aspect of the present application, there is provided an electronic device comprising at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor such that the at least one processor can perform a method according to the first or second aspect.
[0009] According to a sixth aspect of the present application, there is provided a non-transitory computer readable storage medium having computer instructions stored thereon, the computer instructions causing a computer to perform a method according to the first or second aspect.
[0010] According to a seventh aspect of the present application, there is provided a computer program, which, when executed by a processor, realises the method according to the first or second aspect. Effect of the Invention
[0011] According to the technical solution of the present application, cross-modal information (text, layout, and visual) can be used for modeling, and the ability of document understanding can be improved. By using rich text document samples for modeling, the ability of the reading comprehension model to understand rich text documents can be improved. In addition, the present application uses OCR recognition technology instead of information extraction, so there is no need to design a specific information extraction scheme according to the source of the document, making the whole process more end-to-end and saving labor costs.
[0012] However, the contents described in this section are not intended to identify key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present application will become apparent from the following description of the present application. [Brief description of the drawings]
[0013] The drawings are for a better understanding of the present application and are not intended to limit the present disclosure. [Figure 1] FIG. 1 is a schematic diagram according to a first embodiment of the present application. [Diagram 2] FIG. 2 is a schematic diagram according to a second embodiment of the present application. [Diagram 3] FIG. 11 is a schematic diagram according to a third embodiment of the present application. [Figure 4] FIG. 11 is a schematic diagram according to a fourth embodiment of the present application. [Diagram 5] 1 is a schematic diagram of a document reading comprehension model training device based on cross-modal information according to an embodiment of the present application; [Figure 6] FIG. 13 is a schematic diagram of another apparatus for training a document reading comprehension model based on cross-modal information according to an embodiment of the present application. [Figure 7] FIG. 1 is a schematic diagram of a visual question answering apparatus for documents according to an embodiment of the present application; [Figure 8] FIG. 2 is a schematic diagram of another document visual question answering apparatus according to an embodiment of the present application; [Figure 9] FIG. 13 is a schematic diagram of yet another document visual question answering apparatus according to an embodiment of the present application; [Figure 10] FIG. 1 is a block diagram of an electronic device for implementing a document reading comprehension model training method based on cross-modal information or a document visual question answering method according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] Hereinafter, exemplary embodiments of the present application will be described in conjunction with the drawings. In order to facilitate understanding, various details of the embodiments of the present invention are included therein, and they should be regarded as merely examples. Therefore, those skilled in the art can make various changes and modifications to the embodiments described herein without departing from the scope and spirit of the present invention. In addition, for the sake of clarity and conciseness, the following description omits the description of well-known functions and structures.
[0015] 1, which is a flowchart of a method for training a document reading model based on cross-modal information according to a first embodiment of the present application, which is applicable to training a reading model required in a question answering application. As shown in FIG. 1, the method for training a document reading model based on cross-modal information can include the following steps S101 to S104.
[0016] In step S101, a question sample and a rich text document sample are obtained.
[0017] Here, the long rich text document sample includes the actual answers of the question sample. A long rich text document refers to a document that contains more textual information and multiple formats (such as font colors, images, and tables), and its sources include, but are not limited to, web pages, PDFs (Portable Document Formats), and scanned copies of paper documents.
[0018] For example, in response to a question posed by a user using a question and answer application, the question can be obtained as a question sample and a long rich text document corresponding to the question can be obtained as a long rich text document sample, the long rich text document sample including the answer to the question.
[0019] In step S102, an optical character recognition (OCR) process is performed on the image information of the rich text document sample to obtain text information and layout information of the rich text document sample.
[0020] For example, an image capture device can capture image data of a long rich text document sample, and an OCR (Optical Character Recognition) process can be performed on the image data to obtain text and layout information of the long rich text document sample. For example, the long rich text document sample can be considered as a document image, and the document image can be analyzed using OCR recognition technology to obtain the text and layout information.
[0021] In step S103, the text information, layout information and image information of the rich-text document sample are input into a pre-defined reading comprehension model to obtain predicted answers to the question sample.
[0022] For example, the text, layout and image information of a sample rich text document may be input as input data to a reading comprehension model, and the output of the reading comprehension model may be the predicted answer to a sample question.
[0023] In step S104, a reading comprehension model is trained based on the actual answers and the predicted answers.
[0024] For example, based on the difference between the actual answer and the predicted answer, a reading comprehension model can be trained by adjusting relevant parameters of the reading comprehension model to reduce the deviation between the actual answer and the predicted answer.
[0025] By implementing the embodiments of the present application, cross-modal information (text, layout, and visual) can be used for modeling, and the ability of document understanding can be improved. By modeling with rich text document samples, the ability of the reading comprehension model to understand rich text documents can be improved. In addition, the present application uses OCR recognition technology instead of information extraction, so there is no need to design a specific information extraction scheme according to the source of the document, making the whole process more end-to-end and saving labor costs.
[0026] 2, which is a flowchart of a method for training a document reading comprehension model based on cross-modal information according to a second embodiment of the present application, in which the rich text is a long rich text, and the long rich text can be segmented and processed. The method for training a document reading comprehension model based on cross-modal information includes the following steps S201 to S204.
[0027] In step S201, a question sample and a long rich text document sample are obtained.
[0028] Here, the long sample rich text document contains the actual answers to the sample questions.
[0029] In the embodiment of the present application, step S201 can be implemented in any one of the embodiments of the present application, and the embodiment of the present application is not limited thereto, and the description is omitted.
[0030] In step S202, an optical character recognition (OCR) process is performed on the image information of the long rich text document sample to obtain text information and layout information of the long rich text document sample.
[0031] In the embodiment of the present application, step S202 can be implemented in any one of the embodiments of the present application, and the embodiment of the present application is not limited thereto, and the description is omitted.
[0032] In step S203, the text information of the long rich text document sample is divided into a plurality of paragraph texts by a preset step size, and the question sample is spliced with each paragraph text respectively to obtain a plurality of paragraph splice texts.
[0033] For example, the text information of the obtained long rich text document sample is divided into multiple paragraphs in sequence, and each paragraph contains the same number of characters set in advance. For example, taking a preset step size of 512 as an example, the long rich text document sample can be divided into multiple paragraph texts with a step size of 512. Each paragraph is respectively spliced with a question sample to obtain a paragraph splice text.
[0034] In step S204, the multiple paragraph splice texts, layout information and image information are input into a pre-established reading comprehension model to obtain predicted answers for the question sample.
[0035] For example, multiple paragraph splice text, layout information and image information can be input as input data to a reading model, and the output of the reading model can be a predicted answer for a sample question.
[0036] In step S205, a reading comprehension model is trained based on the actual answers and the predicted answers.
[0037] In the embodiment of the present application, step S205 can be implemented in any one of the embodiments of the present application, and the embodiment of the present application is not limited thereto, and the description is omitted.
[0038] By implementing embodiments of the present application, the ability of a reading comprehension model to understand long rich text documents can be improved by segmenting the documents and modeling using the segmented documents.
[0039] Referring to Fig. 3, Fig. 3 is a flowchart of a document reading model training method based on cross-modal information according to a third embodiment of the present application, in which the reading model includes a paragraph encoder, a document encoder, an image encoder and a CRF (Conditional Random Field) module. As shown in Fig. 3, the document reading model training method based on cross-modal information includes the following steps S301 to S309.
[0040] In step S301, a question sample and a long rich text document sample are obtained.
[0041] In the embodiment of the present application, step S301 can be implemented in any one of the embodiments of the present application, and the embodiment of the present application is not limited thereto, and the description is omitted.
[0042] In step S302, an optical character recognition (OCR) process is performed on the image information of the long rich text document sample to obtain text information and layout information of the long rich text document sample.
[0043] In the embodiment of the present application, step S302 can be implemented in any one of the embodiments of the present application, and the embodiment of the present application is not limited thereto, and the description is omitted.
[0044] In step S303, the long rich text document sample is divided into a plurality of paragraph texts by a preset step size, and the question sample is spliced with each paragraph text respectively to obtain a plurality of paragraph spliced texts.
[0045] In the embodiment of the present application, step S303 can be implemented in any one of the embodiments of the present application, and the embodiment of the present application is not limited thereto, and the description is omitted.
[0046] In step S304, the multiple paragraph splice texts, layout information and image information are input into a pre-established reading model.
[0047] In the embodiment of the present application, step S304 can be implemented in any one of the embodiments of the present application, and the embodiment of the present application is not limited thereto, and the description is omitted.
[0048] In step S305, each paragraph splice text and the layout information are encoded by a paragraph encoder to obtain a text code of each paragraph splice text.
[0049] For example, each paragraph splice text and the layout information corresponding to the paragraph splice text can be jointly encoded by a paragraph encoder to generate a text code for each paragraph splice text.
[0050] In step S306, the document encoder performs splicing processing on the text code of each paragraph splice text to obtain the document code of the long rich text document sample.
[0051] For example, a document encoder can splice the text codes of multiple paragraphs of spliced text in order to obtain a document code for a long rich text document sample.
[0052] In step S307, the image information is encoded by an image encoder to obtain visual features of the long rich text document sample.
[0053] Here, in the embodiments of the present application, visual features of a document sample refer to the appearance characteristics of a particular part that are different from other parts of the document, including, but not limited to, bold fonts, changes in font color, underlines, etc.
[0054] For example, an image encoder can obtain visual features of a long rich-text document sample by transforming and combining image information based on preset rules and converting it into digital information data that can be processed by a computer.
[0055] In step S308, a fusion process is performed on the document code and visual features of the long rich text document sample to obtain a multimodal fusion code of the long rich text document sample, and the multimodal fusion code is input into a CRF module to obtain the predicted answer of the question sample.
[0056] For example, a preset feature fusion algorithm model is used to process the document code and visual features of the long rich text document sample to obtain a multimodal fusion code of the long rich text document sample, and the multimodal fusion code is input as input data to a CRF module, and the output data of the CRF module is the predicted answer of the question sample.
[0057] As an example, the feature fusion algorithm model can be a Transformer model, and specific methods of multi-modal fusion may include, but are not limited to, linearly fusing the feature vectors of document codes and visual features, respectively calculating similarity matrices between document codes and visual features and fusing them based on similarity, and directly splicing the feature vectors of document codes and visual features.
[0058] For example, by attaching a sequence label to the multimodal fusion code, the attribution of subsegments is distinguished, answer subfragments of the same attribution are filtered to remove parts that do not belong to the answer fragment, and are integrated with other parts that are not included in the answer subsegments to convert discontinuous answer segments into continuous segments, thereby obtaining answer information corresponding to the question information.
[0059] In step S309, a reading comprehension model is trained based on the actual answers and the predicted answers.
[0060] In the embodiment of the present application, step S309 can be implemented in any one of the embodiments of the present application, and the embodiment of the present application is not limited thereto, and the description is omitted.
[0061] By implementing the embodiments of the present application, processing can be performed based on a question sample and a long rich-text document sample to obtain a multimodal fusion code, and document information can be extracted based on the multimodal fusion code, thereby improving the ability to understand documents and saving labor costs because there is no need to design specific information extraction schemes for documents from various sources.
[0062] In some embodiments of the present application, the reading comprehension model further includes a classifier, and the method for training a document reading comprehension model based on cross-modal information further includes the steps of splicing the multimodal fusion code and the code of the question sample to obtain a splice code, and inputting the splice code into the classifier to obtain a predicted value of the similarity between the question sample and the long rich-text document sample.
[0063] Optionally, the step of training a reading comprehension model based on the actual answers and the predicted answers includes the steps of generating a first loss value based on the actual answers and the predicted answers, generating a second loss value based on the predicted value of similarity and the true value of similarity between the question sample and the long rich text document sample, and training the reading comprehension model based on the first loss value and the second loss value.
[0064] For example, a reading comprehension model can be trained by generating a first loss value based on a true value of the similarity between the actual answer and the predicted answer based on a first preset loss function, incorporating the predicted value of the similarity and the true value of the similarity between the question sample and the long rich text document sample into a second preset loss function to obtain a second loss value, and adjusting related parameters of the reading comprehension model based on the first loss value and the second loss value to reduce the loss value.
[0065] By implementing the embodiment of the present application, the accuracy of answer information generated by the reading comprehension model can be improved by training the reading comprehension model based on the loss value.
[0066] In an optional embodiment, the step of inputting the multimodal fusion code into a CRF module to obtain answer information corresponding to the question information includes the steps of: inputting the multimodal fusion code into a CRF module; obtaining answer sequence labels output from the CRF module, where the answer sequence labels include an answer start label, an answer inner label, an answer outer label, an answer end label, and a single-character answer label; and extracting the answer information corresponding to the question information from the candidate rich text document based on the answer sequence labels.
[0067] The answer sequence labels are obtained by the CRF module by labeling answer sequences from the long rich text document sample based on a sequence labeling scheme of B (Begin) I (Inside) O (outside) E (End) S (Single), where the answer sequence labels include an answer begin label B, an answer inside label I, an answer outside label O, an answer end label E and a single character answer label S.
[0068] Note that since an answer segment may be discontinuous, an answer segment may contain multiple answer sub-segments, and multiple answer sub-segments need to be labeled for subsequent processing, where an answer start label is used to label the starting character position of the answer sub-segment in the answer sequence, a non-answer text label in the answer segment is used to label characters in the answer sub-segment that do not belong to the answer information, an answer external label is used to label characters related to the answer information that are not in the answer sub-segment, an answer end label is used to label the character at the end position of the answer sub-segment in the answer sequence, and a single character answer label is used to label characters that represent separate entities of the answer information.
[0069] Optionally, the answer sequence label further includes non-answer text labels in the answer segment.
[0070] Note that the non-answer text labels in the answer segment are used to label other noise information in the answer sub-segment that does not belong to the answer information, so as to handle the interrupted answer.
[0071] For example, breaks in the same answer text due to discontinuities in the answer information due to tables, page layout, etc. can be labeled.
[0072] Please refer to FIG. 4, which is a flowchart of a visual question answering method for documents according to a third embodiment of the present application. As shown in FIG. 4, the visual question answering method for documents includes the following steps S401 to S404.
[0073] In step S401, in response to the received question information, a candidate rich-text document corresponding to the question information is obtained.
[0074] For example, in response to a question posed by a user received in a question answering application, a rich text document including information related to the question can be obtained and the rich text document can be set as a candidate rich text document.
[0075] In step S402, optical character recognition (OCR) processing is performed on the image information of the candidate rich text document to obtain text information and layout information of the candidate rich text document.
[0076] In the embodiment of the present application, step S402 can be implemented in any one of the embodiments of the present application, and the embodiment of the present application is not limited thereto, and the description is omitted.
[0077] In step S403, the candidate rich text document is divided into a plurality of paragraph texts by a preset step size, and the question information is spliced with each paragraph text to obtain a plurality of paragraph splice texts.
[0078] In the embodiment of the present application, step S403 can be implemented in any one of the embodiments of the present application, and the embodiment of the present application is not limited thereto, and the description is omitted.
[0079] In step S404, answer information corresponding to the question information is generated based on the multiple paragraph splice texts, the layout information, and the image information.
[0080] In one embodiment, the step of generating answer information corresponding to the question information based on the plurality of paragraph splice texts, the layout information and the image information includes the step of inputting the plurality of paragraph splice texts, the layout information and the image information into a pre-established reading comprehension model to obtain answer information corresponding to the question information.
[0081] In the embodiment of the present application, the reading model can be obtained by any one of the embodiments of the present application, and the embodiment of the present application is not limited thereto, and the description will be omitted.
[0082] For example, a plurality of paragraph splice texts, layout information and image information can be input as input data into a pre-established reading comprehension model, and the output data of the model can be the answer information corresponding to the question information.
[0083] In an optional embodiment, the reading comprehension model includes a paragraph encoder, a document encoder, an image encoder, and a CRF module, and the step of inputting a plurality of paragraph splice texts, layout information, and image information into the pre-established reading comprehension model to obtain answer information corresponding to the question information includes: inputting a plurality of paragraph splice texts, layout information, and image information into the reading comprehension model; encoding each paragraph splice text and layout information by the paragraph encoder to obtain a text code of each paragraph splice text; performing a splicing process on the text code of each paragraph splice text by the document encoder to obtain a document code of a candidate rich text document; encoding the image information by the image encoder to obtain visual features of the candidate rich text document; performing a fusion process on the document code and visual features of the candidate rich text document to obtain a multimodal fusion code of the candidate rich text document, and inputting the multimodal fusion code into the CRF module to obtain answer information corresponding to the question information.
[0084] In some embodiments of the present application, the reading comprehension model further comprises a classifier, and the method includes the steps of splicing the multimodal fusion code and the code of the question sample to obtain a splice code, and inputting the splice code to the classifier to obtain a predicted value of the similarity between the question information and the candidate rich-text document.
[0085] Optionally, in response to the predicted value of the similarity being equal to or greater than a threshold, answer information is output.
[0086] For example, in response to a predicted value of the similarity between the question information and the candidate rich text document being equal to or greater than a preset similarity threshold, the generated answer information is output as answer information to the question of the document.
[0087] Optionally, the step of outputting the answer information includes a step of determining coordinate information for each character of the answer information based on the layout information, and a step of highlighting the answer information in the candidate rich text document based on the coordinate information.
[0088] For example, based on the coordinate information, the specific coordinates in the candidate rich text document of each character in the answer information are determined, and based on the coordinate information, the text corresponding to the answer information is highlighted in the candidate rich text document, and after the user opens the candidate rich text document, the user jumps directly to the highlighted answer text.
[0089] Optionally, in response to the predicted similarity being less than a threshold, the answer information is ignored.
[0090] For example, in response to a predicted similarity between the question information and the candidate rich-text document being less than a pre-defined similarity threshold, the generated answer information is ignored.
[0091] By implementing the embodiments of the present application, after answer information is generated, it is possible to determine whether or not to provide the generated answer to the user based on the similarity, thereby suppressing document samples that do not contain answer information and improving the accuracy of the generated answers.
[0092] Optionally, the step of inputting the multimodal fusion code into a CRF module to obtain answer information corresponding to the question information includes the steps of: inputting the multimodal fusion code into a CRF module; obtaining answer sequence labels output from the CRF module, where the answer sequence labels include an answer start label, an answer inner label, an answer outer label, an answer end label and a single-character answer label; and extracting answer information corresponding to the question information from the candidate rich text document based on the answer sequence labels.
[0093] Optionally, the answer sequence label further includes non-answer text labels in the answer segment.
[0094] In the embodiments of the present application, the above steps can be implemented in any one of the embodiments of the present application, and the embodiments of the present application are not limited thereto, and the description is omitted.
[0095] By implementing the embodiments of the present application, the candidate rich-text documents can be modeled, and the documents can be more accurately understood and questions and answers can be made through the pre-trained reading comprehension model based on cross-modal information (text, layout, and visual). At the same time, by using OCR recognition technology instead of information extraction, there is no need to design specific information extraction schemes for documents from various sources, making the whole process more end-to-end and saving labor costs.
[0096] 5, which is a schematic diagram of a document reading model training apparatus based on cross-modal information according to an embodiment of the present application. The document reading model training apparatus based on cross-modal information includes an acquisition module 501, a first processing module 502, a second processing module 503 and a training module 504.
[0097] The acquisition module 501 is used for acquiring a question sample and a rich text document sample, where the rich text document sample includes an actual answer of the question sample; the first processing module 502 is used for performing optical character recognition (OCR) processing on the image information of the rich text document sample to obtain text information and layout information of the rich text document sample; the second processing module 503 is used for inputting the text information, multiple paragraph splice text, layout information and image information of the rich text document sample into a pre-set reading comprehension model to obtain a predicted answer of the question sample; and the training module 504 is used for training the reading comprehension model based on the actual answer and the predicted answer.
[0098] In one embodiment, the rich text is a long rich text, and the second processing module 503 is specifically used to divide the text information of the long rich text document sample according to a preset step size to obtain a plurality of paragraph texts, respectively splice the question sample with each paragraph text to obtain a plurality of paragraph splice texts, and input the plurality of paragraph splice texts, layout information and image information into a preset reading comprehension model to obtain a predicted answer for the question sample. In one embodiment, the reading comprehension model includes a paragraph encoder, a document encoder, an image encoder, and a CRF module. The second processing module 503 specifically inputs a plurality of paragraph splice texts, layout information, and image information into the pre-defined reading comprehension model, encodes each paragraph splice text and layout information through the paragraph encoder to obtain a text code of each paragraph splice text, performs a splicing process on the text code of each paragraph splice text through the document encoder to obtain a document code of the long rich text document sample, encodes the image information through the image encoder to obtain the visual features of the long rich text document sample, performs a fusion process on the document code and the visual features of the long rich text document sample to obtain a multimodal fusion code of the long rich text document sample, and inputs the multimodal fusion code into the CRF module to obtain a predicted answer of the question sample.
[0099] In an optional embodiment, the reading model further comprises a classifier, and the cross-modal information-based document reading model training apparatus further comprises a third processing module. As an example, as shown in Fig. 6, the cross-modal information-based document reading model training apparatus further comprises a third processing module 605 for splicing the multimodal fusion code and the code of the question sample to obtain a splice code, and inputting the splice code into a classifier to obtain a predicted value of the similarity between the question sample and the long rich text document sample. Here, steps S601-S604 in Fig. 6 have the same functions and structures as steps S501-S504 in Fig. 5.
[0100] Optionally, the training module 504 is specifically used to generate a first loss value based on the actual answer and the predicted answer, generate a second loss value based on the predicted similarity value and the true similarity value between the question sample and the long rich text document sample, and train a reading comprehension model based on the first loss value and the second loss value.
[0101] In an optional embodiment, the second processing module 503 specifically inputs the multi-modal fusion code into a CRF module to obtain answer sequence labels output from the CRF module, where the answer sequence labels include an answer start label, an answer inner label, an answer outer label, an answer end label and a single-character answer label, which are used to extract predicted answers for the question sample from the long rich text document sample based on the answer sequence labels.
[0102] Optionally, the answer sequence label further includes non-answer text labels in the answer segment.
[0103] 7, which is a schematic diagram of a visual question answering apparatus for documents according to an embodiment of the present application, the visual question answering apparatus for documents includes an acquisition module 701, a first processing module 702, a second processing module 703, and a generation module 704. Here, the acquisition module 701 is used to acquire a candidate rich text document corresponding to the question information in response to the received question information, the first processing module 702 is used to perform an optical character recognition (OCR) process on the image information of the candidate rich text document to acquire text information and layout information of the candidate rich text document, the second processing module 703 is used to divide the candidate rich text document with a preset step size to acquire a plurality of paragraph texts, and respectively splice the question information with each paragraph text to acquire a plurality of paragraph splice texts, and the generation module 704 is used to generate answer information corresponding to the question information according to the plurality of paragraph splice texts, layout information, and image information.
[0104] In one embodiment, the generation module 704 is specifically used to input multiple paragraph splice texts, layout information and image information into a pre-established reading model to obtain answer information corresponding to question information, where the reading model is trained and obtained by any of the model training methods in the embodiments of the present application.
[0105] In an optional embodiment, the reading comprehension model includes a paragraph encoder, a document encoder, an image encoder, and a CRF module. The generation module 704 specifically inputs multiple paragraph splice texts, layout information, and image information into the reading comprehension model, encodes each paragraph splice text and layout information through the paragraph encoder to obtain a text code of each paragraph splice text, performs a splicing process on the text code of each paragraph splice text through the document encoder to obtain a document code of the candidate rich text document, encodes the image information through the image encoder to obtain the visual features of the candidate rich text document, performs a fusion process on the document code and the visual features of the candidate rich text document to obtain a multimodal fusion code of the candidate rich text document, and inputs the multimodal fusion code into the CRF module to obtain answer information corresponding to the question information.
[0106] Optionally, the reading comprehension model further comprises a classifier, and the document visual question answering apparatus further comprises a third processing module. For example, refer to FIG. 8, as shown in FIG. 8, the document visual question answering apparatus further comprises a third processing module 805 for splicing the multimodal fusion code and the code of the question sample to obtain a splice code, and inputting the splice code into a classifier to obtain a predicted value of the similarity between the question information and the candidate rich text document. Here, steps S801-S804 in FIG. 8 have the same functions and structures as steps S701-S704 in FIG. 7.
[0107] Optionally, the visual question answering apparatus of the document further includes a display module. As an example, referring to Fig. 9, as shown in Fig. 9, the visual question answering apparatus of the document further includes a display module 905 for outputting answer information in response to the predicted value of the similarity being equal to or greater than a threshold. Here, steps S901-S904 in Fig. 9 have the same functions and structures as steps S701-S704 in Fig. 7.
[0108] Optionally, the display module is specifically used to determine coordinate information of each character of the answer information based on the layout information, and highlight the answer information in the candidate rich text document based on the coordinate information.
[0109] Optionally, the generating module 704 specifically inputs the multimodal fusion code into a CRF module, and obtains answer sequence labels output from the CRF module, where the answer sequence labels include an answer start label, an answer inside label, an answer outside label, an answer end label and a single character answer label, and is used to extract answer information corresponding to the question information from the candidate rich text document according to the answer sequence labels. Optionally, the answer sequence labels further include non-answer text labels in the answer segments.
[0110] By implementing the embodiments of the present application, cross-modal information (text, layout, and visual) can be used for modeling, and the ability of document understanding can be improved. By modeling with rich text document samples, the ability of the reading comprehension model to understand rich text documents can be improved. In addition, the present application uses OCR recognition technology instead of information extraction, so there is no need to design a specific information extraction scheme according to the source of the document, making the whole process more end-to-end and saving labor costs.
[0111] For the apparatus in the above embodiment, the specific method by which each module performs the operation is described in detail in the embodiment relating to this method, but will not be described in detail here.
[0112] According to an embodiment of the present application, the present application further provides an electronic device and a readable storage medium. According to an embodiment of the present application, the present application provides a computer program, which, when executed by a processor, realizes a method for training a document reading comprehension model based on cross-modal information or a method for visual question answering of a document provided by an embodiment of the present application.
[0113] As shown in FIG. 10, it is a block diagram of an electronic device of a document reading comprehension model training method based on cross-modal information or a document visual question answering method according to an embodiment of the present application. The electronic device refers to various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, mobile phones, smartphones, wearable devices, and other similar computing devices. The components shown in this specification, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.
[0114] As shown in FIG. 10, the electronic device includes one or more processors 1001, memory 1002, and interfaces for connecting each component with a high-speed interface and a low-speed interface. Each component may be connected by a different bus, implemented on a common motherboard, or implemented in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory for displaying graphical information of a GUI on an external input / output device (e.g., a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used along with multiple memories and multiple memories as needed. Similarly, multiple electronic devices may be connected, each providing a part of the required operations (e.g., a server array, a set of blade servers, or a multi-processor system). In FIG. 10, one processor 1001 is taken as an example.
[0115] The memory 1002 is a non-transitory computer-readable storage medium according to the present application, where the memory stores instructions executable by at least one processor, causing the at least one processor to execute the method for training a document reading comprehension model based on cross-modal information or the method for visual question answering of documents according to the present application. The non-transitory computer-readable storage medium according to the present application stores computer instructions, which are used to execute the method for training a document reading comprehension model based on cross-modal information or the method for visual question answering of documents.
[0116] The memory 1002 can store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions / modules corresponding to the document reading comprehension model training method based on cross-modal information or the document visual question answering method in the embodiments of the present application (e.g., the acquisition module 501, the first processing module 502, the second processing module 503, and the training module 504 shown in FIG. 5, or the third processing module 605 shown in FIG. 6, or the acquisition module 701, the first processing module 702, the second processing module 703, and the generation module 704 shown in FIG. 7, or the third processing module 805 shown in FIG. 8, or the display module 905 shown in FIG. 9) as a non-transitory computer-readable storage medium. The processor 1001 executes the non-transitory software programs, instructions, and modules stored in the memory 1002 to perform various functional applications and data processing of the server, i.e., to implement the document reading comprehension model training method based on cross-modal information or the document visual question answering method in the above-mentioned method embodiments.
[0117] The memory 1002 includes a program storage area capable of storing an operating system, an application required for at least one function, and a data storage area capable of storing data generated based on the use of an electronic device for training a document reading comprehension model based on cross-modal information or visual question and answering of documents. The memory 1002 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1002 may include a memory that is remotely located with respect to the processor 1001, and these remote memories may be connected to the electronic device for training a document reading comprehension model based on cross-modal information or visual question and answering of documents through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0118] The electronic device for the document reading comprehension model training method based on cross-modal information or the document visual question answering method may further include an input device 1003 and an output device 1004. The processor 1001, the memory 1002, the input device 1003, and the output device 1004 may be connected by a bus or other methods, and FIG. 10 illustrates a bus connection as an example.
[0119] The input device 1003 can receive input numeric or character information and generate key signal inputs related to user settings and function control of the electronic device for training a document reading model based on cross-modal information or visual question and answer of documents, and is an input device such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a track ball, a control lever, etc. The output device 1004 may include a display device, an auxiliary lighting device (e.g., LED), and a tactile feedback device (e.g., vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.
[0120] Various embodiments of the systems and techniques described herein may be realized in digital electronic circuitry systems, integrated circuit systems, application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementation in one or more computer programs executable and / or interpretable on a programmable system having at least one programmable processor, which may be a special purpose or general purpose programmable processor, capable of receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
[0121] These computer programs (also referred to as programs, software, software applications, or codes) include machine instructions for a programmable processor, and these computer programs may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, and include a machine-readable medium for receiving machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0122] To provide for interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (e.g., a cathode ray tube (CRT) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices may be used to provide interaction with the user, for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form (including sound input, voice input, or tactile input).
[0123] The systems and techniques described herein may be implemented in a computing system with back-end components (e.g., as a data server), or with middleware components (e.g., an application server), or with front-end components (e.g., a user computer having a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or in a computing system with any combination of such back-end, middleware, or front-end components. The components of the system may be connected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include, for example, a local area network (LAN), a wide area network (WAN), the Internet, etc.
[0124] The computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship is generated by a computer program running on a corresponding computer and having a client-server relationship with each other. The server may be a cloud server, also called a cloud computing server or cloud host, which is a host product in the cloud computing service system, and solves the drawbacks of traditional physical hosts and VPS services (Virtual Private Server, also abbreviated as "VPS"), such as difficult management and poor business scalability. The server may be a distributed system server or a server combined with a blockchain.
[0125] The technical solutions of the embodiments of the present application improve the ability of a document comprehension model to understand long rich-text documents, and eliminate the need to design specific information extraction schemes for documents from various sources, thereby saving labor costs.
[0126] It should be understood that the various forms of flows shown above can be used to change, add, or remove the order of steps. For example, each step described in this application can be performed in parallel, sequentially, or in a different order, and the present specification is not limited thereto as long as the desired results of the technical solutions disclosed in this application can be achieved.
[0127] The above embodiments are not to be construed as limiting the scope of protection of the present application. It is obvious that those skilled in the art can make various modifications, combinations, subcombinations or substitutions according to design requirements and other factors. As long as they do not deviate from the spirit and principle of the present application, any modifications, equivalent replacements and improvements made should be included in the scope of protection of the present application.
Claims
1. A method for training a document reading comprehension model based on cross-modal information, which is executed by a document reading comprehension model training device based on cross-modal information, comprising: obtaining sample questions and sample rich-text documents, the sample rich-text documents including actual answers to the sample questions; performing an optical character recognition (OCR) process on image information of the rich text document sample to obtain text information and layout information of the rich text document sample; inputting the text information, the layout information and the image information of the rich-text document sample into a pre-defined reading comprehension model to obtain a predicted answer for the question sample; training the reading comprehension model based on the actual answers and the predicted answers; Including, the rich text document sample is a long rich text document sample; inputting the text information, the layout information and the image information of the rich-text document sample into a pre-defined reading comprehension model to obtain a predicted answer to the question sample; Dividing the text information of the long rich text document sample by a preset step size to obtain a plurality of paragraph texts, and splicing the question sample with each of the paragraph texts respectively to obtain a plurality of paragraph spliced texts; inputting the plurality of paragraph splice texts, the layout information and the image information into a pre-established reading comprehension model to obtain a predicted answer for the question sample; A document reading comprehension model training method based on cross-modal information, comprising:
2. The reading comprehension model includes a paragraph encoder, a document encoder, an image encoder, and a conditional random field (CRF) module; inputting the plurality of paragraph splice texts, the layout information, and the image information into a pre-established reading comprehension model to obtain a predicted answer for the question sample; inputting the plurality of paragraph splice texts, the layout information and the image information into a pre-established reading model; encoding each of the paragraph splice texts and the layout information by the paragraph encoder to obtain a text code of each of the paragraph splice texts; performing a splicing process on the text code of each of the paragraph splice texts by the document encoder to obtain a document code of the long rich text document sample; encoding the image information by the image encoder to obtain visual features of the long rich text document sample; performing a fusion process on the document code and the visual features of the long rich text document sample to obtain a multi-modal fusion code of the long rich text document sample, and inputting the multi-modal fusion code into the CRF module to obtain a predicted answer of the question sample; The method for training a document reading comprehension model based on cross-modal information according to claim 1 , comprising:
3. the reading comprehension model comprises a classifier; The document reading comprehension model training method based on cross-modal information comprises: splicing the multimodal fusion code and the code of the question sample to obtain a splice code; inputting the splice codes into the classifier to obtain a predicted similarity between the sample question and the sample long rich text document; The method for training a document reading comprehension model based on cross-modal information according to claim 2, comprising:
4. training the reading comprehension model based on the actual answers and the predicted answers; generating a first loss value based on the actual response and the predicted response; generating a second loss value based on the predicted similarity value and the true similarity value between the sample query and the sample long rich text document; training the reading comprehension model based on the first loss value and the second loss value; The method for training a document reading comprehension model based on cross-modal information according to claim 3, comprising:
5. inputting the multimodal fusion code into the CRF module to obtain predicted answers to the sample questions; inputting the multimodal fusion code into the CRF module; obtaining answer sequence labels output from the CRF module, the answer sequence labels including an answer start label, an answer inside label, an answer outside label, an answer end label and a single character answer label; extracting predicted answers for the sample questions from the sample long rich-text document based on the answer sequence labels; The method for training a document reading comprehension model based on cross-modal information according to claim 2, comprising:
6. The method for training a document reading comprehension model based on cross-modal information according to claim 5 , wherein the answer sequence labels include non-answer text labels in answer segments.
7. A method for visual question answering of a document executed by a visual question answering apparatus for a document, comprising: in response to the received query information, obtaining candidate rich-text documents corresponding to the query information; performing an optical character recognition (OCR) process on image information of the candidate rich text document to obtain text information and layout information of the candidate rich text document; Dividing the candidate rich text document by a preset step size to obtain a plurality of paragraph texts, and splicing the question information with each of the paragraph texts respectively to obtain a plurality of paragraph splice texts; generating answer information corresponding to the question information based on the plurality of paragraph splice texts, the layout information, and the image information; Including, generating answer information corresponding to the question information based on the plurality of paragraph splice texts, the layout information, and the image information; A visual question-answering method for documents, comprising a step of inputting the plurality of paragraph splice texts, the layout information and the image information into a pre-established reading comprehension model to obtain answer information corresponding to the question information, wherein the pre-established reading comprehension model is the reading comprehension model obtained by training using the cross-modal information based document reading model training method described in claim 1.
8. The reading comprehension model includes a paragraph encoder, a document encoder, an image encoder, and a conditional random field (CRF) module; inputting the plurality of paragraph splice texts, the layout information and the image information into a pre-established reading comprehension model to obtain answer information corresponding to the question information; inputting the plurality of paragraph splice texts, the layout information and the image information into the reading model; encoding each of the paragraph splice texts and the layout information by the paragraph encoder to obtain a text code of each of the paragraph splice texts; performing a splicing process on the text code of each of the paragraph splice texts by the document encoder to obtain a document code of the candidate rich text document; encoding the image information by the image encoder to obtain visual features of the candidate rich text document; performing a fusion process on the document code and the visual features of the candidate rich text document to obtain a multi-modal fusion code of the candidate rich text document, and inputting the multi-modal fusion code into the CRF module to obtain answer information corresponding to the question information; 8. The method of claim 7, further comprising:
9. The reading comprehension model comprises a classifier, and the document visual question answering method comprises: splicing the multimodal fusion code and the code of the question sample to obtain a splice code; inputting the splice code into the classifier to obtain a predicted similarity between the question information and the candidate rich-text document; 9. The method of claim 8, further comprising:
10. 10. The method for visual question answering of documents according to claim 9, further comprising the step of outputting the answer information in response to the predicted value of the similarity being equal to or greater than a threshold value.
11. The step of outputting the answer information includes: determining coordinate information of each character of the answer information based on the layout information; highlighting the answer information in the candidate rich-text document based on the coordinate information; The method of claim 10, further comprising:
12. 11. The method of claim 10, further comprising the step of ignoring the answer information in response to the predicted similarity being less than the threshold.
13. inputting the multimodal fusion code into the CRF module to obtain answer information corresponding to the question information; inputting the multimodal fusion code into the CRF module; Obtaining answer sequence labels output from the CRF module, the answer sequence labels including an answer start label, an answer inside label, an answer outside label, an answer end label and a single character answer label; extracting answer information corresponding to the question information from the candidate rich-text documents based on the answer sequence labels; 9. The method of claim 8, further comprising:
14. The method of claim 13 , wherein the answer sequence labels include non-answer text labels within answer segments.
15. a capture module for capturing sample questions and sample rich-text documents, the sample rich-text documents including actual answers to the sample questions; a first processing module for performing an optical character recognition (OCR) process on image information of the rich text document sample to obtain text information and layout information of the rich text document sample; a second processing module for inputting text information, the layout information, and the image information of the rich-text document sample into a pre-defined reading comprehension model to obtain a predicted answer for the question sample; a training module for training the reading comprehension model based on the actual responses and the predicted responses; Equipped with the rich text document sample is a long rich text document sample; The second processing module: Divide the text information of the long rich text document sample into a predetermined step size to obtain a plurality of paragraph texts, and respectively splice the question sample into each of the paragraph texts to obtain a plurality of paragraph spliced texts; The apparatus for training a document reading comprehension model based on cross-modal information inputs the plurality of paragraph splice texts, the layout information and the image information into a pre-established reading comprehension model to obtain a predicted answer to the question sample.
16. The reading comprehension model includes a paragraph encoder, a document encoder, an image encoder, and a conditional random field (CRF) module; The second processing module: inputting the plurality of paragraph splice texts, the layout information and the image information into a pre-established reading model; Encoding each of the paragraph splice texts and the layout information by the paragraph encoder to obtain a text code of each of the paragraph splice texts; performing a splicing process on the text code of each of the paragraph splice texts by the document encoder to obtain a document code of the long rich text document sample; encoding the image information by the image encoder to obtain visual features of the long rich text document sample; The cross-modal information-based document reading comprehension model training device of claim 15, further comprising: performing a fusion process on the document code and the visual features of the long rich text document sample to obtain a multi-modal fusion code of the long rich text document sample; and inputting the multi-modal fusion code into the CRF module to obtain a predicted answer of the question sample.
17. the reading comprehension model comprises a classifier; The document reading comprehension model training device based on the cross-modal information, 17. The apparatus for training a document reading comprehension model based on cross-modal information of claim 16, further comprising a third processing module for splicing the multimodal fusion code and a code of the question sample to obtain a splice code, and inputting the splice code to the classifier to obtain a predicted value of similarity between the question sample and the long rich text document sample.
18. The training module comprises: generating a first loss value based on the actual response and the predicted response; generating a second loss value based on the predicted similarity value and the true similarity value between the sample query and the sample long rich text document; The apparatus for training a document reading model based on cross-modal information according to claim 17, wherein the reading model is trained based on the first loss value and the second loss value.
19. The second processing module: Entering the multimodal fusion code into the CRF module; Obtain answer sequence labels output from the CRF module, the answer sequence labels including an answer start label, an answer inside label, an answer outside label, an answer end label and a single character answer label; The apparatus for training a document reading comprehension model based on cross-modal information according to claim 16, further comprising: extracting predicted answers to the question sample from the long rich-text document sample based on the answer sequence labels.
20. The apparatus for training a document reading comprehension model based on cross-modal information according to claim 19, wherein the answer sequence label includes a non-answer text label in an answer segment.
21. an acquisition module responsive to the received query information to acquire candidate rich-text documents corresponding to the query information; a first processing module for performing an optical character recognition (OCR) process on image information of the candidate rich text document to obtain text information and layout information of the candidate rich text document; a second processing module for dividing the candidate rich text document into a plurality of paragraph texts by a predetermined step size, and splicing the question information with each of the paragraph texts to obtain a plurality of paragraph splice texts; a generation module that generates answer information corresponding to the question information based on the plurality of paragraph splice texts, the layout information, and the image information; Equipped with The generation module: A visual question answering device for documents, comprising: inputting the plurality of paragraph splice texts, the layout information, and the image information into a pre-established reading comprehension model to obtain answer information corresponding to the question information, the pre-established reading comprehension model being the reading comprehension model obtained by training using a document reading model training device based on cross-modal information described in any one of claims 15 to 20.
22. The reading comprehension model includes a paragraph encoder, a document encoder, an image encoder, and a conditional random field (CRF) module; The generation module: inputting the plurality of paragraph splice texts, the layout information and the image information into the reading model; Encoding each of the paragraph splice texts and the layout information by the paragraph encoder to obtain a text code of each of the paragraph splice texts; performing a splicing process on the text code of each of the paragraph splice texts by the document encoder to obtain a document code of the candidate rich text document; encoding the image information by the image encoder to obtain visual features of the candidate rich text document; The document visual question answering device of claim 21, further comprising: performing a fusion process on the document code of the candidate rich text document and the visual features to obtain a multimodal fusion code of the candidate rich text document; and inputting the multimodal fusion code into the CRF module to obtain answer information corresponding to the question information.
23. the reading comprehension model comprises a classifier; The visual question answering apparatus for the document, 23. The visual question answering apparatus for documents of claim 22, further comprising a third processing module that splices the multimodal fusion code and a code of the question sample to obtain a splice code, and inputs the splice code to the classifier to obtain a predicted value of similarity between the question information and the candidate rich text document.
24. 24. The visual question answering apparatus for documents according to claim 23, further comprising a display module that outputs the answer information in response to the predicted value of the similarity being equal to or greater than a threshold.
25. The display module comprises: determining coordinate information of each character of the answer information based on the layout information; 25. The visual question answering apparatus for documents according to claim 24, wherein the answer information is highlighted in the candidate rich text document based on the coordinate information.
26. The generation module: Entering the multimodal fusion code into the CRF module; Obtain an answer sequence label output from the CRF module, the answer sequence label including an answer start label, an answer inside label, an answer outside label, an answer end label and a single character answer label; 23. The visual question answering apparatus for documents according to claim 22, further comprising: extracting answer information corresponding to the question information from the candidate rich text documents based on the answer sequence labels.
27. 27. The visual question answering apparatus of claim 26, wherein the answer sequence labels include non-answer text labels within answer segments.
28. At least one processor; a memory communicatively coupled to the at least one processor; Equipped with An electronic device in which the memory stores a program for causing the at least one processor to execute a document reading comprehension model training method based on cross-modal information described in any one of claims 1 to 6, or a visual question answering method for documents described in any one of claims 7 to 14.
29. A non-transitory computer-readable storage medium, comprising: A non-transitory computer-readable storage medium having stored thereon a computer program for causing a computer to execute a document reading comprehension model training method based on cross-modal information described in any one of claims 1 to 6, or a document visual question answering method described in any one of claims 7 to 14.
30. A computer program causing a processor to execute the method for training a document reading comprehension model based on cross-modal information according to any one of claims 1 to 6, or the method for visual question answering of documents according to any one of claims 7 to 14.
Citation Information
Patent Citations
Method and apparatus for visual question answering, computer device and medium
EP3816818A2
Question answering device, method, and program
JP2014120053A
Method, apparatus and device for quality control and storage medium
US20210326524A1
Learning device, text generating device, learning method, text generating method, and program
WO2021171732A1
