Document understanding model generation method, application method, device, equipment and medium

Abstract

The disclosed embodiments disclose a method for generating a document understanding model, an application method, an apparatus, a device, and a medium. The method includes: determining the training input information required for generating the document understanding model; controlling the document understanding model to perform multimodal information fusion on the document text information, document image information, and document layout information in the training input information; controlling the document understanding model to perform a multimodal joint training task based on the fused information, the multimodal joint training task being used to jointly learn the ability to restore masked text, masked images, and masked layouts; and adjusting the parameters of the document understanding model according to the multimodal joint training task to obtain a converged document understanding model. By utilizing the association between different modalities to jointly learn the document content, document image, and document layout, and utilizing the deep network modeling capability to achieve multimodal deep fusion, the model's fault tolerance for OCR recognition is improved, and the output effect of the document understanding model is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular to a method for generating, applying, apparatus, device, and medium for a document understanding model. Background Art

[0002] Document understanding tasks refer to tasks such as automatically answering questions, detecting modules, and classifying entire pages based on input of document images and text (such as receipts, reports, and ordinary files).

[0003] Most related solutions use a fusion of existing text and images to achieve document understanding. However, these document understanding models lack universality, resulting in poor compatibility with downstream tasks. Furthermore, because document text input is typically generated using an OCR engine, this can introduce significant recognition errors, making these document understanding models sensitive to OCR misalignments. Furthermore, most document understanding approaches use decoder models, requiring modifications to the model code when handling different downstream tasks. Summary of the Invention

[0004] The embodiments of the present disclosure provide a method for generating, an application method, an apparatus, a device, and a medium for a document understanding model to improve the model's tolerance for OCR recognition and improve the output effect of the document understanding model.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for generating a document understanding model, the method comprising:

[0006] Determining training input information required for generating a document understanding model, the training input information including document text information, document image information, and document layout information processed by local masking, the document layout information being used to describe positions of the document text information and document image information in the document;

[0007] Controlling the document understanding model to perform multimodal information fusion on document text information, document image information, and document layout information in the training input information, and controlling the document understanding model to perform a multimodal joint training task based on the fused information, wherein the multimodal joint training task is used to jointly learn the ability to recover masked text, masked image, and masked layout;

[0008] The parameters of the document understanding model are adjusted according to the multimodal joint training task to obtain a converged document understanding model.

[0009] In a second aspect, an embodiment of the present disclosure further provides an application method of a document understanding model, wherein the document understanding model is obtained using the method for generating a document understanding model described in any of the above embodiments, and the method includes:

[0010] Acquiring document information to be processed, wherein the document information to be processed includes document text information, document image information, and document layout information obtained by parsing the document to be processed;

[0011] The document information to be processed is input into the document understanding model to perform document understanding, and a document understanding result of the document to be processed is obtained, wherein the document understanding result includes a document question-answering result, a document detection result and / or a document classification result.

[0012] In a third aspect, an embodiment of the present disclosure further provides a device for generating a document understanding model, the device comprising:

[0013] a training input module, configured to determine training input information required for generating a document understanding model, wherein the training input information includes document text information, document image information, and document layout information processed by local masking, wherein the document layout information is used to describe the location of the document text information and document image information in the document;

[0014] a multimodal training module for controlling the document understanding model to perform multimodal information fusion on document text information, document image information, and document layout information in the training input information, and controlling the document understanding model to perform a multimodal joint training task based on the fused information, wherein the multimodal joint training task is used to jointly learn the ability to recover masked text, masked image, and masked layout;

[0015] A model generation module is used to adjust the parameters of the document understanding model according to the multimodal joint training task to obtain a converged document understanding model.

[0016] In a fourth aspect, an embodiment of the present disclosure further provides an application device for a document understanding model, wherein the document understanding model is obtained using the method for generating a document understanding model described in any of the above embodiments, and the device comprises:

[0017] A document information acquisition module is used to acquire document information to be processed, wherein the document information to be processed includes document text information, document image information, and document layout information obtained by parsing the document to be processed;

[0018] The document understanding module is used to input the information of the document to be processed into the document understanding model to perform document understanding and obtain the document understanding results of the document to be processed, wherein the document understanding results include document question and answer results, document detection results and / or document classification results.

[0019] In a fifth aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0020] at least one processor; and

[0021] a memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for generating a document understanding model or the method for applying a document understanding model described in any one of the above embodiments.

[0023] In a sixth aspect, a computer-readable medium is also provided in an embodiment of the present disclosure, wherein the computer-readable medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for generating a document understanding model or the method for applying a document understanding model described in any one of the above embodiments when executed.

[0024] The technical solution of the embodiment of the present disclosure obtains the training input information required for generating a document understanding model, controls the document understanding model to perform multimodal information fusion on the document text information, document image information and document layout information in the training input information, and controls the document understanding model to perform a multimodal joint training task based on the fused information. The multimodal joint training task is used to jointly learn the ability to restore masked text, masked images and masked layout, adjusts the parameters of the document understanding model according to the multimodal joint training task, introduces training input information of multiple modalities as model training data, utilizes the association between different modalities to jointly learn the document content, document image and document layout, utilizes the deep network modeling capability to realize multimodal deep fusion, improves the recognition fault tolerance capability of the document understanding model in OCR document recognition, and improves the output effect of the document understanding model.

[0025] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0027] Figure 1 This is a flowchart of a method for generating a document understanding model provided by an embodiment of the present disclosure;

[0028] Figure 2 Schematic diagram of the structure and training of a document understanding model applicable to the embodiments of the present disclosure;

[0029] Figure 3 is a flowchart of another method for generating a document understanding model provided by an embodiment of the present disclosure;

[0030] Figure 4 This is a flowchart of a method for applying a document understanding model provided by an embodiment of the present disclosure;

[0031] Figure 5 This is a schematic diagram of the structure of a device for generating a document understanding model provided by an embodiment of the present disclosure;

[0032] Figure 6 This is a schematic diagram of the structure of an application device of a document understanding model provided by an embodiment of the present disclosure;

[0033] Figure 7 It is a structural diagram of an electronic device for implementing a method for generating a document understanding model or a method for applying a document understanding model provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0035] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0036] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0037] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0038] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0039] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0040] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0041] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0042] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0043] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0044] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0045] Figure 1 This is a flowchart of a method for generating a document understanding model provided by an embodiment of the present disclosure. The technical solution of this embodiment can be applied to the execution of document understanding tasks to achieve document question and answer, document detection based on document classification, and the method can be executed by a device for generating a document understanding model. The device can be implemented by software and / or hardware and is generally integrated on any electronic device with network communication capabilities, such as a mobile terminal, PC, or server. Figure 1 As shown, the method for generating a document understanding model in this embodiment may include but is not limited to the following steps S110-S130:

[0046] S110. Determine the training input information required for generating a document understanding model. The training input information includes document text information, document image information, and document layout information processed by local masking. The document layout information is used to describe the positions of the document text information and document image information in the document.

[0047] Document understanding tasks can be performed on document images and text (such as receipts, reports, and general documents) to perform document answering, document detection, and document classification. While the fusion of text and images can achieve document understanding to a certain extent, this approach is not universally applicable. Significant changes to a document template can lead to deviations when using text and images to understand the document.

[0048] To this end, during the pre-training process of the document understanding model, not only document text and document images are used, but also document layout information is introduced. The document text information, document image information, and document layout information that have been processed by local masks are combined into the training data of the document understanding model. The document image can include the overall document image and the local image in the document. The overall document image implies the overall style of the document page, while the local image area in the document implies more text detail features. In this way, the document understanding model can learn not only the overall features of the document but also the detailed features in the document. The document layout information can, to a certain extent, describe the relative position layout of the document text information and the document image information in the document. The document layout information allows the document understanding model to learn the layout features of the document during pre-training, avoiding the subsequent deformation of the document text or document image, which may lead to the inability to perform the document understanding task well.

[0049] S120 , controlling the document understanding model to perform multimodal information fusion on the document text information, document image information, and document layout information in the training input information, and controlling the document understanding model to perform a multimodal joint training task based on the fused information.

[0050] Among them, the multimodal joint training task is used to jointly learn the ability to recover mask text, mask image and mask layout.

[0051] Decoder models are mostly used in document understanding, which means that separate heads are usually required to adapt to different data task forms for different downstream document understanding tasks. For example, a linear classifier layer needs to be added when performing document classification tasks, while a complex detection network needs to be added when performing document detection tasks. These operations require modifications at the model level, and the model design of each downstream document task requires relevant domain knowledge, resulting in high model generation costs.

[0052] To this end, when pre-training the document understanding model, a neural network is used to perform multimodal deep fusion of the document text information, document image information, and document layout information in the training input information, forming a unified input for different downstream document understanding tasks. The unified fused information is then used to control the document understanding model to perform multimodal joint training tasks across modalities. At the same time, the document understanding model is trained to effectively learn the ability to restore masked text content in document text information, restore masked image content in document image information, and restore the position of masked document text information and document image information in the document layout information. This allows the pre-training process of text generation, image generation, and coordinate generation to be completed without changing the model code, greatly improving the development efficiency and portability of the document understanding model.

[0053] As an optional but non-limiting implementation, determining the training input information required for generating the document understanding model may include, but is not limited to, the following steps A1-A3:

[0054] Step A1: Mask the local text content segments in the original document text information parsed from the document to obtain document text information that has been processed with local masking.

[0055] See also Figure 2 , the input of document text is generally obtained using an optical character recognition (OCR) engine, which will introduce more obvious recognition errors, and existing work generally uses an extractive approach to answer questions in documents, which is more sensitive to engine recognition errors. To this end, the document can be obtained and the local text content fragments in the original document text information parsed by OCR from the document can be masked, such as randomly blocking or replacing part of the text content in the original document text information. For example, randomly masking part of the text content in the original document text information results in an average of 30% of the text content being masked. In this way, the masked document text information can be involved in model training, which can improve the document understanding model's fault tolerance for OCR recognition and ensure a higher output effect under the same recognition quality text input conditions.

[0056] Step A2: Masking the local image content segments in the original document image information parsed from the document to obtain document image information processed with local masking.

[0057] See also Figure 2In addition to the document text input that may be obtained using the OCR engine, the document image input may also be obtained using a suitable engine, and parsing deviations may also occur. To this end, the local image content fragments in the original document image information parsed from the document can be masked, such as randomly blocking part of the image content in the original document image information. For example, randomly masking part of the image content in the original document image information causes an average of 50% of the image content to be masked. In this way, the masked document image information can participate in model training, which can improve the document understanding model's fault tolerance for image recognition and ensure a higher output effect under the image input conditions of the same recognition quality.

[0058] Step A3: Mask the positions of the local text content segments and / or local image content segments in the original document layout information parsed from the document to obtain document layout information that has been locally masked.

[0059] Similar to the masking process of the document text information and document image information described above, the relative positions of the local text content segments and / or local image content segments in the original document layout information parsed from the document are also randomly masked. For example, the positions of the local text content segments and / or local image content segments in the original document layout information are randomly masked so that an average of 20% of the positions are masked. In this way, the masked document layout information can be used in model training, which can improve the document understanding model's tolerance for layout recognition and ensure a higher output effect under the same recognition quality layout input conditions.

[0060] In the above method, through a unified masking method, after completing the masking of the training input information of various modalities in the document, the model can use the training input information that has been uniformly masked to perform the model training process, ensuring that the model can be better migrated to different downstream document understanding tasks.

[0061] S130. Adjust the parameters of the document understanding model according to the multimodal joint training task to obtain a converged document understanding model.

[0062] The technical solution of the embodiment of the present disclosure controls the document understanding model to perform multimodal information fusion on the document text information, document image information and document layout information in the training input information, and controls the document understanding model to perform multimodal joint training tasks based on the fused information. By introducing training input information of multiple modalities as model training data, the association between different modalities is used to jointly learn the document content, document image and document layout, and the deep network modeling capability is used to achieve multimodal deep fusion, thereby improving the recognition fault tolerance of the document understanding model during OCR recognition and improving the output effect of the document understanding model.

[0063] Figure 3 This is a flowchart of another method for generating a document understanding model provided by an embodiment of the present disclosure. The technical solution of this embodiment further optimizes the process of multimodal information fusion of document text information, document image information and document layout information in the training input information in the above embodiment. This embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 3 As shown, the method for generating a document understanding model in this embodiment may include the following steps S310-S370:

[0064] S310. Determine the training input information required for generating a document understanding model. The training input information includes document text information, document image information, and document layout information processed by local masking. The document layout information is used to describe the positions of the document text information and document image information in the document.

[0065] S320. Control the document understanding model to encode the document text information in the training input information to obtain a target document text vector.

[0066] S330. Control the document understanding model to encode the document image information in the training input information to obtain a target document image vector.

[0067] S340: Control the document understanding model to encode the document layout information in the training input information to obtain a target document layout vector.

[0068] See also Figure 2 The document understanding model may include a first embedding block for encoding and generating a vector. After obtaining the document text information, document image information and document layout information processed by local masking as training input information, the document text information, document image information and document layout information can be input into the first embedding block respectively, and the corresponding target document text vector, target document image vector and target document layout vector are output through the first embedding block.

[0069] See also Figure 2For text modality, the text content in the document text information can be input into the text encoding network of the first embedding block of the document understanding model, and the input text content is encoded into the corresponding target document text vector through the first embedding block. For image modality, the image pixels in the document image information can also be input into the image encoding network corresponding to the first embedding block, and the image pixels corresponding to the document image information are encoded through the image encoding network to obtain the target document image vector. For layout modality, the text content position and image content position in the document layout information can be input into the layout encoding network of the first embedding block in the document understanding model, and the input text content position and image content position are encoded into the corresponding target document layout vector through the first embedding block. The coordinate range of the text content and image content area in the document corresponding to the document layout vector in the document page can be represented by a position box.

[0070] S350 , performing vector fusion on the target document text vector, the target document image vector, and the target document layout vector to obtain fused information obtained through multimodal information fusion.

[0071] As an optional but non-limiting implementation, performing vector fusion on the target document text vector, the target document image vector, and the target document layout vector may include steps B1-B2:

[0072] Step B1: concatenate the target document text vector and the target document image vector and perform position encoding.

[0073] Step B2: Add the target document layout vector and the vector obtained after position encoding to complete multimodal information fusion.

[0074] See also Figure 2 , the target document text vector corresponding to the document text information and the target document image vector corresponding to the document image information are concatenated. The document understanding model can also include a second embedding block for position encoding. This second embedding block can positionally encode the concatenation result of the target document text vector and the target document image vector, thereby preventing the self-attention mechanism in the document understanding model from being unable to recognize the position information of the text and image, and preventing the text and image from being shuffled, resulting in consistent attention weights. The target document layout vector is then added to the vector obtained after position encoding to complete the multimodal information fusion process, which serves as input for controlling the document understanding model to perform multimodal joint training tasks.

[0075] In the above optional methods, in order to solve the problem of not being able to perform joint training for all modalities, document layout is introduced for pre-training on the basis of pre-training with text and images. By adding embedding blocks to the document understanding model, the information of different modalities is encoded and processed separately and then fused, ensuring that the subsequent multimodal joint training has a unified input. Through the multimodal pre-training of unified document understanding, all downstream document understanding tasks are unified in form, and text generation, image generation and coordinate generation are completed without changing the model code, which significantly improves the effect of downstream document understanding tasks.

[0076] S360. Based on the fused information, the document understanding model is controlled to perform a multimodal joint training task, where the multimodal joint training task is used to jointly learn the ability to recover masked text, masked images, and masked layout.

[0077] As an optional but non-limiting implementation, controlling the document understanding model to perform a multimodal joint training task based on the fused information may include but is not limited to the following steps C1-C3:

[0078] Step C1: Based on the fused information, the document understanding model is controlled to perform a text prediction training task. The training goal of the text prediction training task is to predict the local text content in the document text information that is obscured by the mask.

[0079] Step C2: Based on the fused information, the document understanding model is controlled to perform an image prediction training task. The training goal of the image prediction training task is to predict the local image content obscured by the mask in the document image information in order to restore the document image.

[0080] Step C3: Based on the fused information, the document understanding model is controlled to perform a layout prediction training task. The training goal of the layout prediction training task is to predict the position of the local text content segment and / or the position of the local image content segment that is obscured by the mask in the document layout information.

[0081] The multimodal joint training tasks performed by the document understanding model can include text prediction training tasks, image prediction training tasks, and layout prediction training tasks. The text prediction training tasks, image prediction training tasks, and layout prediction training tasks are self-supervised training tasks.

[0082] See also Figure 2The document understanding model includes a multimodal encoder block and decoder block based on a self-attention mechanism. When training the document understanding model, the encoder and decoder in the document understanding model unify the training paradigms of different modalities and use a unified mask-prediction method to control the document understanding model to perform text prediction training tasks, image prediction training tasks, and layout prediction training tasks respectively. The document understanding model is trained with unsupervised data, achieving unified cross-modal training of the document understanding model and ensuring that the document understanding model is adaptable to different downstream document understanding tasks.

[0083] See also Figure 2 For the text modality, when the document understanding model is controlled based on the fused information to perform the text prediction training task, local text content fragments can be randomly masked in the document text information of the fused information, for example, an average of 30% of the text content is masked. The document understanding model is trained through the text prediction training task to predict the masked text content, and the masked text content is refilled into the document text information.

[0084] See also Figure 2 For the image modality, when the document understanding model is controlled based on the fused information to perform the image prediction training task, local image content fragments can be randomly masked in the document image information of the fused information, for example, so that an average of 50% of the image content is masked. Through the image prediction training task, the document understanding model is trained to have the ability to predict the masked image content and restore the complete document image information.

[0085] See also Figure 2 For the layout mode, when the document understanding model is controlled based on the fused information to perform the layout prediction training task, the local text content segment positions and / or local image content segment positions in the document layout information of the fused information are masked, such as randomly masking 20% ​​of the local text content segments and / or local image content segment positions. The document understanding model is trained through the layout prediction training task to improve its prediction ability for the masked local text content segment positions and / or local image content segment positions, and the masked local text content segment positions and / or local image content segment positions are restored and output.

[0086] By adopting the above optional method, different downstream document understanding tasks are unified into one input, and then the unified fused information is used to control the document understanding model to perform multimodal joint training tasks across modalities. At the same time, the document understanding model is trained to perform unsupervised learning on the ability to recover masked text content in document text information, masked image content in document image information, and the position of masked document text information and document image information in document layout information, significantly improving the output effect of downstream document understanding tasks. By utilizing a series of new technologies in deep learning, especially in the fields of natural language processing and image processing, such as the use of transformer models, unsupervised pre-training, and multimodal information fusion in document understanding model training, good output results have been achieved in a series of downstream document understanding tasks.

[0087] As an optional but non-limiting implementation, controlling the document understanding model to perform the multimodal joint training task based on the fused information may also include the following process:

[0088] When controlling the document understanding model to perform multimodal joint training tasks, the relative position representations between text content and image content in the fused information are enhanced in the encoder block of the document understanding model based on the decoupled attention mechanism, so as to enhance the importance of relative positions in the multimodal joint training tasks.

[0089] See also Figure 2 In the encoding block of the document understanding model, a disentangled attention mechanism is introduced to enhance the importance of the relative positions between the text content and the image content in the fused information in the multimodal joint training task. In this way, when performing the multimodal joint training task in the decoding block of the document understanding model, the relative positions between the text content and the image content in the fused information can be fully utilized for learning. The above method significantly improves the output effect of the document understanding model and improves the understanding accuracy of the document understanding model.

[0090] As an optional but non-limiting implementation, controlling the document understanding model to perform the multimodal joint training task based on the fused information may also include the following process:

[0091] When controlling the document understanding model to perform multimodal joint training tasks, the training tasks of different modalities during multimodal joint training are modally separated in the decoder block of the document understanding model based on the hybrid expert mechanism to reduce the mutual interference between different modal training tasks.

[0092] See also Figure 2In the decoding block of the document understanding model, the Mixture of Expert (MoE) mechanism is introduced. Through the Mixture of Expert (MoE) mechanism, the training tasks of different modalities can be separated when the decoding block performs multimodal joint training, which solves the problem of mutual interference between different modalities during multimodal joint training. Experiments have shown that it has a certain improvement on the output effect of the document understanding model and can further improve the understanding accuracy of the document understanding model.

[0093] S370. Adjust the parameters of the document understanding model according to the multimodal joint training task to obtain a converged document understanding model.

[0094] As an optional but non-limiting implementation, adjusting the parameters of the document understanding model according to the multimodal joint training task may include the process of steps D1-D2:

[0095] Step D1: Obtain a pre-trained document understanding model by performing a multimodal joint training task.

[0096] In the process of performing multimodal joint training tasks on the document understanding model, the loss value when performing the multimodal joint training tasks can be calculated using the loss function of the document understanding model based on the input and output of the decoder of the document understanding model when performing the multimodal joint training tasks, and then the parameters of the document understanding model are updated based on the calculated loss value to obtain a pre-trained document understanding model in the pre-training stage.

[0097] Optionally, the document understanding model includes a multimodal encoder block and a decoder block based on a self-attention mechanism. The loss function used by the document understanding model to perform the multimodal joint training task can be:

[0098]

[0099] in, The unified loss function used to perform multimodal joint training tasks; T is the total number of pre-training training tasks; N i is the maximum length of the task i sequence; ω i is the weight of training task i; x' i is the masked input of task i; task i can be a text prediction training task, an image prediction training task, or a layout prediction training task; and y are the predicted target output of the decoder of the document understanding model and the input of the decoder of the document understanding model respectively.

[0100] Alternatively, see Figure 2Considering that the graphic content of the document image information is a whole, a separate image encoder can be trained using VQ-VAE image encoding technology. The image pixels in the document image information are input into the trained image encoder to obtain a discrete image. In this way, the image content fragments that are masked in the document image information can be labeled, so that the loss calculation can be performed based on the label participation of the masked part in the document image information.

[0101] Step D2: Start the training tasks that match the downstream document understanding task among the at least two modal training tasks included in the pre-trained document understanding model in the fine-tuning stage, and fine-tune the parameters of the pre-trained document understanding model through the started training tasks to obtain a converged document understanding model that matches the downstream document understanding task.

[0102] As an optional but non-restrictive implementation method, the training tasks of various modalities in the document understanding model are completed in the pre-training stage through a unified mask-prediction method, so that the model can be better migrated to different downstream document understanding tasks. At the same time, the document understanding model can also adopt an instruction mechanism based on natural language. The instruction mechanism is used to instruct the document understanding model on the training objectives of each modality training task when performing multimodal joint training tasks, so as to reduce the difference between the document understanding model in the pre-training stage and the document understanding model in the fine-tuning stage. For example, instructions about the task objectives are added before the document input. For example: in the training task of the pre-training stage of the text modality, the instruction mechanism is: "What is the complete text for <mask>tokens?" Experiments have shown that the closer the instructions of the downstream document understanding task are to the instructions during pre-training, the better the effect of the pre-trained document understanding model is and the more it matches the downstream document understanding task.

[0103] Alternatively, for pre-training and fine-tuning the document understanding model, the same pre-training dataset as the existing model was used, and the pre-trained model was fine-tuned on the downstream dataset. This achieved state-of-the-art results on both the DocVQA question-answering task and the CORD sequence labeling task. Competitive results were also achieved on the Publaynet object detection task and the RVL-CDIP document classification task. Using the same quality text input, the DocVQA question-answering task significantly outperformed the state-of-the-art results, demonstrating the model's strong tolerance for OCR input errors.

[0104] The technical solution of the embodiment of the present disclosure controls the document understanding model to perform multimodal information fusion on the document text information, document image information and document layout information in the training input information, and controls the document understanding model to perform multimodal joint training tasks based on the fused information. By introducing training input information of multiple modalities as model training data, the document content, document image and document layout are jointly learned using the association between different modalities, and the deep network modeling capability is used to achieve multimodal deep fusion, thereby improving the recognition fault tolerance of the document understanding model during OCR document recognition and improving the output effect of the document understanding model. At the same time, cross-modal unified training is achieved to ensure that the document understanding model can adapt to different downstream document understanding tasks. By adopting a unified generative model, the forms of all downstream document understanding tasks are unified, and text generation, image generation and coordinate generation can be completed without changing the model code, which greatly improves development efficiency and portability. It has been verified through experiments that the fault tolerance for OCR recognition can be improved. Under the text input conditions of the same recognition quality, the output effect is significantly higher than that of existing methods.

[0105] Figure 4 This is a flowchart of a method for applying a document understanding model provided by an embodiment of the present disclosure. The technical solution of this embodiment can be applied to situations where document understanding tasks are performed to achieve document question-answering and document detection based on document classification. This method can be performed by an application device for a document understanding model. The device can be implemented by software and / or hardware and is generally integrated into any electronic device with network communication capabilities, such as a mobile terminal, PC, or server. Figure 4 As shown, the application method of the document understanding model of this embodiment may include but is not limited to the following steps S410-S420:

[0106] S410: Acquire information of a document to be processed, where the information includes document text information, document image information, and document layout information obtained by parsing the document to be processed.

[0107] The document understanding model is obtained by using any of the aforementioned methods for generating a document understanding model.

[0108] S420: Input the information of the document to be processed into the document understanding model to perform document understanding, and obtain a document understanding result for the document to be processed, wherein the document understanding result includes a document question-and-answer result, a document detection result, and / or a document classification result.

[0109] Figure 5 This is a structural block diagram of a device for generating a document understanding model provided by an embodiment of the present disclosure. The technical solution of this embodiment can be applied to situations where document understanding tasks are performed to achieve document question-answering and document detection based on document classification. The device can be implemented by software and / or hardware and is generally integrated into any electronic device with network communication capabilities, such as a mobile terminal, PC, or server. Figure 5 As shown, the document understanding model generation device of this embodiment may include: a training input module 510, a multimodal training module 520 and a model generation module 530. Among them:

[0110] A training input module 510 is configured to determine training input information required for generating a document understanding model, wherein the training input information includes document text information, document image information, and document layout information processed by local masking, wherein the document layout information is used to describe the location of the document text information and document image information in the document;

[0111] a multimodal training module 520 for controlling the document understanding model to perform multimodal information fusion on the document text information, document image information, and document layout information in the training input information, and controlling the document understanding model to perform a multimodal joint training task based on the fused information, wherein the multimodal joint training task is used to jointly learn the ability to recover masked text, masked image, and masked layout;

[0112] The model generation module 530 is used to adjust the parameters of the document understanding model according to the multimodal joint training task to obtain a converged document understanding model.

[0113] Based on the above embodiment, optionally, determining the training input information required for generating the document understanding model includes:

[0114] Masking a local text content segment in the original document text information parsed from the document to obtain document text information processed by local masking;

[0115] Masking a local image content segment in the original document image information parsed from the document to obtain document image information processed by local masking;

[0116] The positions of the local text content segments and / or the local image content segments in the original document layout information parsed from the document are masked to obtain document layout information that has been subjected to local masking.

[0117] Based on the above embodiment, optionally, controlling the document understanding model to perform multimodal information fusion on document text information, document image information, and document layout information includes:

[0118] Controlling the document understanding model to encode the document text information in the training input information to obtain a target document text vector;

[0119] Controlling the document understanding model to encode the document image information in the training input information to obtain a target document image vector;

[0120] Controlling the document understanding model to encode the document layout information in the training input information to obtain a target document layout vector;

[0121] The target document text vector, the target document image vector and the target document layout vector are fused to obtain fused information obtained by multimodal information fusion.

[0122] Based on the above embodiment, optionally, the target document text vector, the target document image vector, and the target document layout vector are fused, including:

[0123] The target document text vector and the target document image vector are concatenated and position encoded;

[0124] The target document layout vector is added to the vector obtained after position encoding to complete the multimodal information fusion.

[0125] Based on the above embodiment, optionally, controlling the document understanding model to perform a multimodal joint training task based on the fused information includes:

[0126] Controlling the document understanding model to perform a text prediction training task based on the fused information, wherein the training objective of the text prediction training task is to predict the local text content obscured by the mask in the document text information;

[0127] Controlling the document understanding model to perform an image prediction training task based on the fused information, wherein the training objective of the image prediction training task is to predict local image content obscured by the mask in the document image information so as to restore the document image;

[0128] Controlling the document understanding model to perform a layout prediction training task based on the fused information, wherein the training objective of the layout prediction training task is to predict the position of a local text content segment and / or a local image content segment that is obscured by a mask in the document layout information;

[0129] Among them, the text prediction training task, the image prediction training task and the layout prediction training task are self-supervised training tasks.

[0130] Based on the above embodiment, optionally, controlling the document understanding model to perform a multimodal joint training task based on the fused information further includes:

[0131] When controlling the document understanding model to perform a multimodal joint training task, the representation of the relative positions between text contents and image contents in the fused information is enhanced in the encoder block of the document understanding model based on the decoupled attention mechanism, so as to enhance the importance of the relative positions in the multimodal joint training task.

[0132] Based on the above embodiment, optionally, controlling the document understanding model to perform a multimodal joint training task based on the fused information further includes:

[0133] When controlling the document understanding model to perform multimodal joint training tasks, the training tasks of different modalities during multimodal joint training are modally separated in the decoder block of the document understanding model based on a hybrid expert mechanism to reduce mutual interference between different modal training tasks.

[0134] Based on the above embodiment, optionally, the document understanding model adopts an instruction mechanism based on natural language, which is used to indicate the training objectives of each modality training task when performing a multimodal joint training task, so as to reduce the difference between the document understanding model in the pre-training stage and the document understanding model in the fine-tuning stage.

[0135] Based on the above embodiment, optionally, adjusting the parameters of the document understanding model according to the multimodal joint training task includes:

[0136] Obtain a pre-trained document understanding model by performing multimodal joint training tasks;

[0137] The training tasks of at least two modalities included in the pre-trained document understanding model that match the downstream document understanding task are started in the fine-tuning stage, and the parameters of the pre-trained document understanding model are fine-tuned through the started training tasks.

[0138] See also Figure 2 The document model uses multimodal fusion technology to adapt to multiple inputs, and simultaneously fuses multimodal document text information, document image information, and document layout information, unifying the input of different modal training in multimodal joint training. It also designs modal separation technology to better adapt to any modality as output in text prediction training tasks, image prediction training tasks, and layout prediction training tasks. Corresponding pre-training tasks are designed for different modalities and the training paradigm is unified. After pre-training the document understanding model, only a small amount of supervised training is required for the pre-trained document understanding model based on the downstream document understanding task to obtain a document understanding model that meets the requirements of the downstream document understanding task.

[0139] Based on the above embodiment, optionally, the document understanding model includes a first embedding block for encoding and generating a vector, a second embedding block for position encoding, and a multimodal encoder block and decoder block based on a self-attention mechanism.

[0140] The device for generating a document understanding model provided in the embodiments of the present disclosure can execute the method for generating a document understanding model provided in any of the above embodiments of the present disclosure, and has the corresponding functions and beneficial effects of executing the method for generating a document understanding model. For detailed processes, please refer to the relevant operations of the method for generating a document understanding model in the above embodiments.

[0141] Figure 6 This is a structural block diagram of an application device for a document understanding model provided by an embodiment of the present disclosure. The technical solution of this embodiment can be applied to situations where document understanding tasks are performed to achieve document question-answering and document detection based on document classification. The device can be implemented by software and / or hardware and is generally integrated into any electronic device with network communication capabilities, such as a mobile terminal, PC, or server. Figure 4 As shown, the application device of the document understanding model of this embodiment may include: a document information acquisition module 610 and a document understanding module 620. Among them:

[0142] The document information acquisition module 610 is used to acquire the document information to be processed, where the document information to be processed includes document text information, document image information, and document layout information obtained by parsing the document to be processed.

[0143] The document understanding model is obtained using any of the aforementioned methods for generating the document understanding model. When applied, the document understanding model uses an auto-regressive approach to generate the required document question-and-answer results, document detection results, and / or document classification results from left to right.

[0144] The document understanding module 620 is used to input the document information to be processed into the document understanding model to perform document understanding and obtain the document understanding result of the document to be processed, wherein the document understanding result includes a document question and answer result, a document detection result and / or a document classification result.

[0145] The application device of the document understanding model provided in the embodiments of the present disclosure can execute the application method of the document understanding model provided in any of the above embodiments of the present disclosure, and has the corresponding functions and beneficial effects of executing the application method of the document understanding model. For detailed processes, please refer to the relevant operations of the application method of the document understanding model in the above embodiments.

[0146] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0147] Figure 7 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 7 , which shows an electronic device (eg Figure 7 The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0148] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the electronic device 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An edit / output (I / O) interface 705 is also connected to the bus 704.

[0149] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0150] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method for generating a document understanding model or the method for applying a document understanding model shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method for generating a document understanding model or the method for applying a document understanding model of the embodiment of the present disclosure are executed.

[0151] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0152] The electronic device provided by the embodiment of the present disclosure and the method for generating a document understanding model or the method for applying a document understanding model provided by the above-mentioned embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above-mentioned embodiment, and this embodiment has the same beneficial effects as the above-mentioned embodiment.

[0153] An embodiment of the present disclosure provides a computer storage medium having a computer program stored thereon. When the program is executed by a processor, the method for generating a document understanding model or the method for applying a document understanding model provided in the above embodiment is implemented.

[0154] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0155] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0156] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0157] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0158] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: determines the training input information required for generating a document understanding model, the training input information includes document text information, document image information and document layout information processed by local masking, and the document layout information is used to describe the position of the document text information and document image information in the document; controls the document understanding model to perform multimodal information fusion on the document text information, document image information and document layout information in the training input information, and controls the document understanding model to perform a multimodal joint training task based on the fused information, and the multimodal joint training task is used to jointly learn the ability to restore masked text, masked image and masked layout; adjusts the parameters of the document understanding model according to the multimodal joint training task to obtain a converged document understanding model.

[0159] Alternatively, the computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains document information to be processed, wherein the document information to be processed includes document text information, document image information and document layout information obtained by parsing the document to be processed; inputs the document information to be processed into the document understanding model for document understanding, and obtains a document understanding result for the document to be processed, wherein the document understanding result includes a document question and answer result, a document detection result and / or a document classification result.

[0160] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0161] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0162] The units described in the embodiments of the present disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0163] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0164] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0165] According to one or more embodiments of the present disclosure, Example 1 provides a method for generating a document understanding model, the method comprising:

[0166] Determining training input information required for generating a document understanding model, the training input information including document text information, document image information, and document layout information processed by local masking, the document layout information being used to describe positions of the document text information and document image information in the document;

[0167] Controlling the document understanding model to perform multimodal information fusion on document text information, document image information, and document layout information in the training input information, and controlling the document understanding model to perform a multimodal joint training task based on the fused information, wherein the multimodal joint training task is used to jointly learn the ability to recover masked text, masked image, and masked layout;

[0168] The parameters of the document understanding model are adjusted according to the multimodal joint training task to obtain a converged document understanding model.

[0169] According to one or more embodiments of the present disclosure, Example 2 determines the training input information required for generating a document understanding model according to the method of Example 1, including:

[0170] Masking a local text content segment in the original document text information parsed from the document to obtain document text information processed by local masking;

[0171] Masking a local image content segment in the original document image information parsed from the document to obtain document image information processed by local masking;

[0172] The positions of the local text content segments and / or the local image content segments in the original document layout information parsed from the document are masked to obtain document layout information that has been subjected to local masking.

[0173] According to one or more embodiments of the present disclosure, Example 3, based on the method of Example 1, controls the document understanding model to perform multimodal information fusion on document text information, document image information, and document layout information, including:

[0174] Controlling the document understanding model to encode the document text information in the training input information to obtain a target document text vector;

[0175] Controlling the document understanding model to encode the document image information in the training input information to obtain a target document image vector;

[0176] Controlling the document understanding model to encode the document layout information in the training input information to obtain a target document layout vector;

[0177] The target document text vector, the target document image vector and the target document layout vector are subjected to vector fusion to obtain fused information obtained through multimodal information fusion.

[0178] According to one or more embodiments of the present disclosure, Example 4, according to the method of Example 3, performs vector fusion on the target document text vector, the target document image vector, and the target document layout vector, including:

[0179] The target document text vector and the target document image vector are concatenated and position encoded;

[0180] The target document layout vector is added to the vector obtained after position encoding to complete the multimodal information fusion.

[0181] According to one or more embodiments of the present disclosure, Example 5, based on the method of Example 1, controls the document understanding model to perform a multimodal joint training task based on the fused information, including:

[0182] Controlling the document understanding model to perform a text prediction training task based on the fused information, wherein the training objective of the text prediction training task is to predict the local text content obscured by the mask in the document text information;

[0183] Controlling the document understanding model to perform an image prediction training task based on the fused information, wherein the training objective of the image prediction training task is to predict local image content obscured by the mask in the document image information so as to restore the document image;

[0184] Controlling the document understanding model to perform a layout prediction training task based on the fused information, wherein the training objective of the layout prediction training task is to predict the position of a local text content segment and / or a local image content segment that is obscured by a mask in the document layout information;

[0185] Among them, the text prediction training task, the image prediction training task and the layout prediction training task are self-supervised training tasks.

[0186] According to one or more embodiments of the present disclosure, Example 6, according to the method of Example 5, controls the document understanding model to perform a multimodal joint training task based on the fused information, further comprising:

[0187] When controlling the document understanding model to perform a multimodal joint training task, the representation of the relative positions between text contents and image contents in the fused information is enhanced in the encoder block of the document understanding model based on the decoupled attention mechanism, so as to enhance the importance of the relative positions in the multimodal joint training task.

[0188] According to one or more embodiments of the present disclosure, Example 7, according to the method of Example 5, controls the document understanding model to perform a multimodal joint training task based on the fused information, further comprising:

[0189] When controlling the document understanding model to perform multimodal joint training tasks, the training tasks of different modalities during multimodal joint training are modally separated in the decoder block of the document understanding model based on a hybrid expert mechanism to reduce mutual interference between different modal training tasks.

[0190] According to one or more embodiments of the present disclosure, Example 8 is based on the method described in Example 1, and the document understanding model adopts an instruction mechanism based on natural language, and the instruction mechanism is used to indicate the training objectives of each modality training task when performing a multimodal joint training task, so as to reduce the difference between the document understanding model in the pre-training stage and the document understanding model in the fine-tuning stage.

[0191] According to one or more embodiments of the present disclosure, Example 9, based on the method of Example 1, adjusts the parameters of the document understanding model according to the multimodal joint training task, including:

[0192] The pre-trained document understanding model is obtained by performing multimodal joint training tasks in the pre-training phase;

[0193] The training tasks of at least two modalities included in the pre-trained document understanding model that match the downstream document understanding task are started in the fine-tuning stage, and the parameters of the pre-trained document understanding model are fine-tuned through the started training tasks.

[0194] According to one or more embodiments of the present disclosure, Example 10 is based on the method described in any one of Examples 1-9, and the document understanding model includes a first embedding block for encoding and generating a vector, a second embedding block for position encoding, and a multimodal encoder block and decoder block based on a self-attention mechanism.

[0195] According to one or more embodiments of the present disclosure, Example 11 provides a method for applying a document understanding model, wherein the document understanding model is obtained using the method for generating a document understanding model described in any one of Examples 1-10, and the method includes:

[0196] Acquiring document information to be processed, wherein the document information to be processed includes document text information, document image information, and document layout information obtained by parsing the document to be processed;

[0197] The document information to be processed is input into the document understanding model to perform document understanding, and a document understanding result of the document to be processed is obtained, wherein the document understanding result includes a document question-answering result, a document detection result and / or a document classification result.

[0198] According to one or more embodiments of the present disclosure, Example 12 provides a device for generating a document understanding model, the device comprising:

[0199] a training input module, configured to determine training input information required for generating a document understanding model, wherein the training input information includes document text information, document image information, and document layout information processed by local masking, wherein the document layout information is used to describe the location of the document text information and document image information in the document;

[0200] a multimodal training module for controlling the document understanding model to perform multimodal information fusion on document text information, document image information, and document layout information in the training input information, and controlling the document understanding model to perform a multimodal joint training task based on the fused information, wherein the multimodal joint training task is used to jointly learn the ability to recover masked text, masked image, and masked layout;

[0201] A model generation module is used to adjust the parameters of the document understanding model according to the multimodal joint training task to obtain a converged document understanding model.

[0202] According to one or more embodiments of the present disclosure, Example 13 provides an application device for a document understanding model, wherein the document understanding model is obtained using the method for generating a document understanding model described in any one of Examples 1-10, and the device includes:

[0203] A document information acquisition module is used to acquire document information to be processed, wherein the document information to be processed includes document text information, document image information, and document layout information obtained by parsing the document to be processed;

[0204] The document understanding module is used to input the information of the document to be processed into the document understanding model to perform document understanding and obtain the document understanding results of the document to be processed, wherein the document understanding results include document question and answer results, document detection results and / or document classification results.

[0205] According to one or more embodiments of the present disclosure, Example 14 provides an electronic device, the electronic device including:

[0206] at least one processor; and

[0207] a memory communicatively connected to the at least one processor; wherein,

[0208] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for generating a document understanding model described in any one of Examples 1-10 or the method for applying a document understanding model described in Example 11.

[0209] According to one or more embodiments of the present disclosure, Example 15 provides a computer-readable medium, which stores computer instructions, and the computer instructions are used to enable a processor to implement the method for generating a document understanding model described in any one of Examples 1-10 or the method for applying a document understanding model described in Example 11 when executed.

[0210] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0211] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0212] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.< / mask>

Claims

1. A method for generating a document understanding model, characterized in that: The method comprises: Determining training input information required for generating a document understanding model, the training input information including document text information, document image information, and document layout information processed by local masking, the document layout information being used to describe the location of the document text information and document image information in the document; the document layout information enabling the document understanding model to learn document layout features during pre-training; Controlling the document understanding model to perform multimodal information fusion on the document text information, document image information, and document layout information in the training input information, and controlling the document understanding model to perform a multimodal joint training task based on the fused information, wherein the multimodal joint training task is used to jointly learn the ability to recover masked text, masked image, and masked layout; the encoder block of the document understanding model is used to encode the information of different modalities in the training input information separately and then fuse them; wherein controlling the document understanding model to perform the multimodal joint training task based on the fused information further includes: when controlling the document understanding model to perform the multimodal joint training task, performing modal separation on the training tasks of different modalities during the multimodal joint training in the decoder block of the document understanding model based on a hybrid expert mechanism; The parameters of the document understanding model are adjusted according to the multimodal joint training task to obtain a converged document understanding model.

2. The method according to claim 1, characterized in that Identify the training input information needed to generate the document understanding model, including: Masking a local text content segment in the original document text information parsed from the document to obtain document text information processed by local masking; Masking a local image content segment in the original document image information parsed from the document to obtain document image information processed by local masking; The positions of the local text content segments and / or the local image content segments in the original document layout information parsed from the document are masked to obtain document layout information that has been subjected to local masking.

3. The method according to claim 1, characterized in that Controlling the document understanding model to perform multimodal information fusion on document text information, document image information, and document layout information, including: Controlling the document understanding model to encode the document text information in the training input information to obtain a target document text vector; Controlling the document understanding model to encode the document image information in the training input information to obtain a target document image vector; Controlling the document understanding model to encode the document layout information in the training input information to obtain a target document layout vector; The target document text vector, the target document image vector and the target document layout vector are fused to obtain fused information obtained by multimodal information fusion.

4. The method according to claim 3, characterized in that Performing vector fusion on the target document text vector, the target document image vector, and the target document layout vector, including: The target document text vector and the target document image vector are concatenated and position encoded; The target document layout vector is added to the vector obtained after position encoding to complete the multimodal information fusion.

5. The method according to claim 1, characterized in that Controlling the document understanding model to perform a multimodal joint training task based on the fused information includes: Controlling the document understanding model to perform a text prediction training task based on the fused information, wherein the training objective of the text prediction training task is to predict the local text content obscured by the mask in the document text information; Controlling the document understanding model to perform an image prediction training task based on the fused information, wherein the training objective of the image prediction training task is to predict the local image content obscured by the mask in the document image information; Controlling the document understanding model to perform a layout prediction training task based on the fused information, wherein the training objective of the layout prediction training task is to predict the position of a local text content segment and / or a local image content segment that is obscured by a mask in the document layout information; Among them, the text prediction training task, the image prediction training task and the layout prediction training task are self-supervised training tasks.

6. The method according to claim 5, characterized in that Controlling the document understanding model to perform a multimodal joint training task based on the fused information also includes: When controlling the document understanding model to perform a multimodal joint training task, the representation of the relative positions between text contents and the relative positions between image contents in the fused information is enhanced in the encoder block of the document understanding model based on a decoupled attention mechanism.

7. The method according to claim 1, characterized in that The document understanding model adopts an instruction mechanism based on natural language, which is used to indicate the training objectives of each modality training task when performing a multimodal joint training task.

8. The method according to any one of claims 1 to 7, characterized in that: The document understanding model includes a first embedding block for encoding and generating a vector, a second embedding block for position encoding, and a multimodal encoder block and decoder block based on a self-attention mechanism.

9. A method for applying a document understanding model, characterized in that: The document understanding model is obtained by using the method for generating a document understanding model according to any one of claims 1 to 8, the method comprising: Acquiring document information to be processed, wherein the document information to be processed includes document text information, document image information, and document layout information obtained by parsing the document to be processed; The document information to be processed is input into the document understanding model to perform document understanding, and a document understanding result of the document to be processed is obtained, wherein the document understanding result includes a document question-answering result, a document detection result and / or a document classification result.

10. A device for generating a document understanding model, characterized in that: The device comprises: A training input module is configured to determine training input information required for generating a document understanding model, wherein the training input information includes document text information, document image information, and document layout information processed by local masking, wherein the document layout information is used to describe the location of the document text information and document image information in the document; the document layout information enables the document understanding model to learn document layout features during pre-training; A multimodal training module controls the document understanding model to perform multimodal information fusion on the document text information, document image information, and document layout information in the training input information, and controls the document understanding model to perform a multimodal joint training task based on the fused information, wherein the multimodal joint training task is used to jointly learn the ability to recover masked text, masked image, and masked layout; the encoder block of the document understanding model is used to encode the information of different modalities in the training input information separately and then fuse them; wherein, controlling the document understanding model to perform the multimodal joint training task based on the fused information further includes: when controlling the document understanding model to perform the multimodal joint training task, performing modal separation on the training tasks of different modalities during the multimodal joint training in the decoder block of the document understanding model based on a hybrid expert mechanism; A model generation module is used to adjust the parameters of the document understanding model according to the multimodal joint training task to obtain a converged document understanding model.

11. An application device of a document understanding model, characterized in that: The document understanding model is obtained by using the method for generating a document understanding model according to any one of claims 1 to 8, and the device includes: A document information acquisition module is used to acquire document information to be processed, wherein the document information to be processed includes document text information, document image information, and document layout information obtained by parsing the document to be processed; The document understanding module is used to input the information of the document to be processed into the document understanding model to perform document understanding and obtain the document understanding results of the document to be processed, wherein the document understanding results include document question and answer results, document detection results and / or document classification results.

12. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for generating a document understanding model described in any one of claims 1 to 8 or the method for applying a document understanding model described in claim 9.

13. A computer-readable medium, characterized in that The computer-readable medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for generating a document understanding model according to any one of claims 1 to 8 or the method for applying a document understanding model according to claim 9 when executed.

Images