A method and device for implementing a cross-modal document pre-training model

Abstract

The application relates to the field of information technology and provides a cross-modal document pre-training model implementation method and device.The purpose is to solve the problem that LayoutLMv3 cannot focus on a task, causing tasks to affect each other and leading to unsatisfactory performance of an ongoing task.The main scheme comprises the following steps: obtaining pre-training document image data;adding 2D text position embedding, 1D text position embedding and masked text information embedding, and taking the addition result as a to-be-fused text embedding vector;adding 2D image position embedding, 1D image position embedding and masked image information embedding, and taking the addition result as a to-be-fused image embedding vector;connecting the to-be-fused image embedding vector and the to-be-fused text embedding vector to obtain a multi-modal fusion embedding vector for pre-training model training; and through the pre-training model, different pre-training models are selected according to different task categories to perform fine-tuning training on different data sets, so that a model meeting a corresponding task is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology, and provides a cross-modal document pre-training model implementation method and device. BACKGROUND

[0002] A large number of works show that the pre-training model on a large corpus can learn a general language representation, which is beneficial to downstream natural language processing tasks, and can avoid training a new model from scratch. Document intelligence is a rapidly developing research topic and actual industrial demand in recent years, mainly refers to the text and rich layout format information contained in web pages, digital documents or scanned documents, and the process of understanding, classification, extraction and information induction through artificial intelligence technology. Due to the diversity of layout and format, low-quality scanned document images and the complexity of template structure, document intelligence is a very challenging task and has attracted extensive attention in the related field. For document-level tasks, the overall document image can provide global structural information. For example, the overall document structure of a personal resume and the document structure of a scientific literature are obviously different in vision. The visual features exhibited by these modality-aligned rich text formats can be extracted by a visual model and combined into the pre-training phase, thereby effectively helping downstream tasks.

[0003] LayoutLMv3, a technology developed by Microsoft Asia Research Institute, unifies various document information recognition, extraction, and processing tasks. It uses unified text and image masks for document information pre-training. During the pre-training phase, LayoutLMv3 combines text information and image information in the document for mask learning. Mask learning is a self-supervised technique that has made rapid progress in representation learning. In natural language processing research, the transformer-based bidirectional encoder representation technique first proposed "masked language modeling" by predicting the original vocabulary of randomly masked word tokens based on their context to learn bidirectional representations. Although most multi-modal pre-training document models use the text modality MLM proposed by the transformer-based bidirectional encoder representation technique, they differ in the pre-training objectives of the image modality. For example, DocFormer proposes to regress masked region features, which are noisier and more difficult to learn than classifying discrete features in a smaller vocabulary. The different granularity of image (dense image pixels or continuous region features) and text (discrete tokens) objectives further increases the difficulty of cross-modal alignment learning, which is crucial for multi-modal representation learning. LayoutLMv3 directly utilizes the original image patches in the document image without the need for complex preprocessing steps such as page object detection. LayoutLMv3 jointly learns image, text, and multi-modal representations in a Transformer model with unified masked language modeling, masked image modeling, and word patch alignment objectives. This makes LayoutLMv3 the first multi-modal pre-training document intelligence model without convolutional neural networks for image embedding, significantly saving parameters and freeing from region annotation.

[0004] This simple unified architecture and objective make LayoutLMv3 a general pre-training model for text-centric tasks and image-centric document intelligence tasks.

[0005] Although LayoutLMv3's approach is one of the better-performing techniques in cross-modal document pre-training models, it has some obvious drawbacks that this proposal addresses.

[0006] First, LayoutLMv3's first drawback is that the model itself has a very large number of parameters, requiring 368 million parameter adjustments for each model training and model inference. This large number of parameters makes the model unable to be directly used in actual application scenarios. In real life, when users want to use document intelligence in their work (such as real-time correction of word, grammar, and punctuation errors, financial report analysis, etc.), they do not want to wait for an excessively long time.

[0007] Secondly, LayoutLMv3 does not focus on text and image tasks during pre-training, and the training method for two different tasks is a training method, which leads to the fact that LayoutLMv3 may perform financial report analysis when performing real-time correction word tasks, which will affect different tasks and lead to unsatisfactory performance of the ongoing task. SUMMARY

[0008] The purpose of the present application is to solve the problem that LayoutLMv3 cannot focus on tasks, leading to mutual influence between tasks and unsatisfactory performance of the ongoing task.

[0009] In order to solve the above technical problems, the present application adopts the following technical means:

[0010] The present application provides a cross-modal document pre-training model implementation method, comprising the following steps:

[0011] Step 1, obtaining pre-training document image data;

[0012] Step 2, the document image is recognized as document text information by a character recognition method, obtaining document text content and corresponding 2D text position information and 1D text position information, wherein the 2D text position information is the boundary box coordinates of the text sequence, and the 1D text position information is the number of rows and position offset information in the image block (i.e. each page) where the text is located.

[0013] The document text content obtains text information embedding through a trained vocabulary, and the document image information is directly used as image information embedding;

[0014] The 2D text position information and the 1D text position information obtain 2D text position embedding and 1D text position embedding through a position encoding function, wherein the position encoding function can be defined by the user or the commonly used position encoding function publicly available on the Internet.

[0015] The 2D image position information is the boundary box coordinates of the image block;

[0016] The 1D image position information is the sequential numbering of the image block from left to right and from top to bottom, wherein the initial numbering is 1;

[0017] The 2D image position information and the 1D image position information obtain 2D image position embedding and 1D image position embedding through a position encoding function;

[0018] Step 3, adjust the document image to HxW, and represent the image in the dimension of CxHxW, where C, H and W are the channel size, width and height of the image respectively, then split the image into a series of uniform PxP size blocks, linearly project the image blocks to the dimension and flatten them into vectors, that is, obtain the embedding representation of the image information;

[0019] Step 4, first, when performing text information embedding and image information embedding, random mask processing is needed for the two kinds of embedding vectors to obtain masked text information embedding and masked image information embedding;

[0020] Step 4.1, add the 2D text position embedding, the 1D text position embedding and the masked text information embedding, and the result of the addition is used as the to-be-fused text embedding vector;

[0021] Step 4.2, add the 2D image position embedding, the 1D image position embedding and the masked image information embedding, and the result of the addition is used as the to-be-fused image embedding vector;

[0022] The above two strategies can make the text and image parts absorb each other's position information, which can make the position information completely match the original information, and can affect the adjustment result of the weight in the subsequent matrix multiplication operation, filter out important text and image components, and finally improve the overall performance.

[0023] Step 4.3, connect the to-be-fused image embedding vector and the to-be-fused text embedding vector to obtain a multi-modal fusion embedding vector;

[0024] Step 5, pre-training model training, the multi-modal fusion embedding vector obtained in step 4.3 is used as the input of the initial network, different training target weights are used for different multi-modal document tasks, and a word patch alignment target is used to learn the fine-grained alignment between text words and image blocks. The word patch alignment target is to predict whether the corresponding image block of the text word is masked, and a word patch alignment loss is obtained. The word patch alignment loss, the text loss and the image loss are added respectively to obtain the total loss of the model;

[0025] Step 6, through the pre-training model, different pre-training models are selected according to different task categories to perform fine-tuning training on different data sets, and a fine-tuning model corresponding to the corresponding task is obtained;

[0026] Step 7, use the fine-tuning model to perform the corresponding document task.

[0027] In the above technical solution, different training target weights are used to perform different pre-training for text-centered document intelligent tasks and image-centered document intelligent tasks, wherein:

[0028] The text-centered document intelligent task training target is a masked sequence context representation based on image embedding and text embedding, the model maximizes the probability of correctly predicting the text token, and finally obtains the loss of the text-centered pre-training model;

[0029] The image-centered document intelligent task is driven by cross-entropy loss to reconstruct the masked image token x in the context of its surrounding text and image tokens, and the label of the image token x comes from an image tokenizer which can convert dense image pixels into discrete tokens according to a visual vocabulary, and finally obtains the loss of the image-centered pre-training model.

[0030] The application provides a cross-modal document pre-training model implementation device, comprising the following modules:

[0031] The data acquisition module obtains pre-trained document image data;

[0032] The extraction module identifies the document image into document text information through a character recognition method to obtain document text content and corresponding 2D text position information and 1D text position information, and simultaneously obtains 2D image position information and 1D image position information;

[0033] The embedding module adjusts the document image to HxW and represents the image in the dimension of CxHxW, wherein C, H and W are respectively the channel size, width and height of the image, then splits the image into a series of uniform PxP size blocks, linearly projects the image blocks to the dimension and flattens them into vectors, and finally obtains the embedding representation of the image information;

[0034] The multi-modal fusion module first needs to perform random mask processing on the two embedding vectors when performing text information embedding and image information embedding to obtain masked text information embedding and masked image information embedding;

[0035] Step 4.1, add the 2D text position embedding, the 1D text position embedding and the masked text information embedding, and the addition result is used as a to-be-fused text embedding vector;

[0036] Step 4.2, add the 2D image position embedding, the 1D image position embedding and the masked image information embedding, and the addition result is used as a to-be-fused image embedding vector;

[0037] Step 4.3, connect the to-be-fused image embedding vector and the to-be-fused text embedding vector to obtain a multi-modal fusion embedding vector;

[0038] The loss calculation module, the pre-training model training, the multi-modal fusion embedding vector obtained in step 4.3 is used as the input of the initial network, different training target weights are used for different multi-modal document tasks, a word patch alignment target is used to learn the fine-grained alignment between text words and image blocks, the word patch alignment target is to predict whether the corresponding image block of the text word is masked, a word patch alignment loss is obtained, and the word patch alignment loss is added to the text loss and the image loss to obtain a total loss of the model;

[0039] The training module, by using the pre-training model, different pre-training models are selected according to different task categories to perform fine-tuning training on different data sets to obtain a fine-tuning model corresponding to the task;

[0040] The fine-tuning module, using the fine-tuning model to perform the corresponding document task.

[0041] In the above device, in the extraction module, the 2D text position information is the boundary box coordinates of the text sequence, and the 1D text position information is the row number and position offset information of the text in the image block;

[0042] The document text content obtains text information embedding through a trained word table, and the document image information is directly used as image information embedding;

[0043] The 2D text position information and the 1D text position information obtain 2D text position embedding and 1D text position embedding through a position encoding function, wherein the position encoding function can be defined by the user himself or a commonly used position encoding function on the Internet.

[0044] In the above extraction module,

[0045] The 2D image position information is the boundary box coordinates of the image block;

[0046] The 1D image position information is the sequential number from left to right and from top to bottom of the image block, wherein the initial number is 1;

[0047] The 2D image position information and the 1D image position information obtain 2D image position embedding and 1D image position embedding through a position encoding function.

[0048] In the above device, different training target weights are used to perform different pre-training for the text-centered document intelligent task and the image-centered document intelligent task, wherein:

[0049] The training target of the text-centered document intelligent task is a masked sequence context representation based on image embedding and text embedding, the model maximizes the probability of correctly predicting the text label, and finally obtains the loss of the text-centered pre-training model;

[0050] The image-centered document intelligent task is driven by a cross-entropy loss to reconstruct the masked image token x in the context of the surrounding text and image tokens, and the label of the image token x is from an image tokenizer which can convert dense image pixels into discrete tokens according to a visual vocabulary, and finally obtain the loss of the image-centered pre-training model.

[0051] Because the application adopts the above technical scheme, the following beneficial effects are achieved:

[0052] 1. The application proposes a cross-modal document pre-training model that integrates text information and image information, which can better intelligently process document information.

[0053] 2. Although the parameter quantity and LayOutLMv3 are very close in the pre-training stage, the application only needs to modify a small part of the parameters to obtain better results during the downstream task fine-tuning. In addition, the application uses a distillation technology for model compression, so that the model inference stage is more in line with the speed of real life.

[0054] 3. The application pre-trains different modal-centered tasks in the pre-training stage, respectively text-centered and image-centered, which can better match the downstream tasks. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 The flowchart of the application;

[0056] Figure 2 The block diagram of constructing image embedding and text embedding;

[0057] Figure 3 The block diagram of mask processing;

[0058] Figure 4 The block diagram of modal embedding processing and fusion. DETAILED DESCRIPTION

[0059] The embodiments of the application will be described in detail below. Although the application will be described and illustrated by means of some specific embodiments, it is to be understood that the application is not restricted to just these embodiments. Rather, any modification or equivalent replacement thereto should be included in the scope of the claims of the application.

[0060] In addition, in order to better illustrate the application, numerous specific details are given in the specific embodiments below. Those skilled in the art will understand that the application can be implemented without these specific details.

[0061] The application provides a cross-modal document pre-training model implementation method, comprising the following steps:

[0062] Step 1: Obtain pre-training document image data

[0063] The document image data required in the pre-training stage is massive, so that the model can learn the general representation of various document tasks. Unlike the strict and labeled data set required by traditional document recognition tasks, the document image data in the pre-training stage only needs document images.

[0064] These data can be artificially constructed or obtained from the Internet. The present technical solution takes the large open source data set IIT-CDIP as an example: this data set is a large-scale scanned document image data set, which contains about 11 million document images.

[0065] Step 2: The document image is recognized as document text information by a character recognition method, obtaining document text content and corresponding 2D text location information, and 1D text location information, wherein the 2D text location information is the boundary box coordinates of the text sequence, and the 1D text location information is the number of lines and position offset information in the image block (that is, each page) where the text is located.

[0066] The document text content obtains text information embedding through the trained vocabulary, and the document image information is directly used as image information embedding;

[0067] The 2D text location information and the 1D text location information obtain 2D text location embedding and 1D text location embedding through a position encoding function, wherein the position encoding function can be defined by the user or the commonly used position encoding function publicly available on the Internet.

[0068] The 2D image location information is the boundary box coordinates of the image block;

[0069] The 1D image location information is the sequential numbering of the image block from left to right and from top to bottom, wherein the initial numbering is 1;

[0070] The 2D image location information and the 1D image location information obtain 2D image location embedding and 1D image location embedding through a position encoding function;

[0071] Step 3: Linear projection features are used as image information

[0072] Most pre-training models for document rely on object detectors like Faster R-CNN to extract local information of image embeddings, which is inefficient. Inspired by LayoutLMv3, the same way is used to extract image information: linear projection features are used as the feature representation of image information. The linear projection feature representation of the image block is represented as the document image, and then fed into the multi-modal Transformer. Specifically, the document image is adjusted to HxW, and the image is represented in the dimension of CxHxW, where C, H and W are the channel size, width and height of the image, respectively. Then the image is split into a series of uniform Pxp blocks, and the image block is linearly projected to dimension and flattened into a vector (similar to word vectors), i.e. the embedding representation of image information.

[0073] Step 4: First, when performing text information embedding and image information embedding, random mask processing needs to be performed on the two embedding vectors to obtain masked text information embedding and masked image information embedding.

[0074] Step 4.1, add 2D text position embedding, 1D text position embedding and masked text information embedding, and the result of addition is used as the to-be-fused text embedding vector.

[0075] Step 4.2, add 2D image position embedding, 1D image position embedding and masked image information embedding, and the result of addition is used as the to-be-fused image embedding vector.

[0076] Step 4.3, the to-be-fused image embedding vector and the to-be-fused text embedding vector are connected to obtain a multi-modal fusion embedding vector.

[0077] Step 5: Pre-training model training

[0078] The multi-modal fusion embedding vector obtained in step 4 is used as the input of the initial recognition network, and different training target weights are used for different multi-modal document tasks. Therefore, in order to focus on different modal tasks, the system respectively pre-trains the text-centered document intelligent task and the image-centered document intelligent task.

[0079] The training target of the text-centered document intelligent task is the masked sequence context representation based on image embedding and text embedding, and the model maximizes the probability of correctly predicting the text label. Finally, the loss of the text-centered pre-training model is obtained.

[0080] The image-centric document intelligence task is driven by a cross-entropy loss to reconstruct the masked image tokens x under the context of the surrounding text and image tokens. The labels of the image tokens come from an image tokenizer that can convert dense image pixels into discrete tokens according to a visual vocabulary, and finally obtain the loss of the image-centric pre-training model.

[0081] In addition, the system uses a word patch alignment target to learn the fine-grained alignment between text words and image blocks, as LayoutLMv3 does. The word patch alignment target is to predict whether the corresponding image block of the text word is masked, and the word patch alignment loss is obtained. The text loss and the image loss are added respectively to obtain the total loss of the model. If the total loss of the model or the current iteration number meets the model convergence condition, the current initial model is taken as the pre-training model, otherwise, the network parameters of the initial recognition network are adjusted based on the total loss of the model to obtain an updated initial model. The sample fusion features of the document image are input into the updated initial model for feature extraction and prediction of the mask feature, so as to perform iterative training of the updated initial model until the total loss of the model or the iteration number meets the model convergence condition, and the pre-training recognition network is obtained. Wherein, the model convergence condition can be that the total loss is less than or equal to a preset loss, or the iteration number reaches a preset number.

[0082] In addition, in the case that the importance of the document and the importance of the image are the same, the above three losses are all added to the pre-training model for training.

[0083] Step 6: Through the pre-training model, different pre-training models are selected according to different task categories to perform fine-tuning training on different data sets, and a fine-tuning model conforming to the corresponding task is obtained.

[0084] Step 7: Use the fine-tuning model to perform the corresponding document task.

[0085] The application provides a document pre-training model implementation device supporting cross-modal, comprising the following modules:

[0086] The data acquisition module obtains pre-training document image data;

[0087] The extraction module identifies the document image into document text information by a character recognition method to obtain document text content and corresponding 2D text position information and 1D text position information, and simultaneously obtains 2D image position information and 1D image position information;

[0088] The embedding module adjusts the document image to HxW and represents the image in the dimension of CxHxW, wherein C, H and W are respectively the channel size, width and height of the image, then the image is split into a series of uniform PxP size blocks, the image blocks are linearly projected to the dimension of and flattened into a vector, and the embedding representation of the image information is obtained.

[0089] The multi-modal fusion module first needs to perform random mask processing on the two kinds of embedding vectors when performing text information embedding and image information embedding to obtain masked text information embedding and masked image information embedding.

[0090] Step 4.1, adding the 2D text position embedding, the 1D text position embedding and the masked text information embedding, and taking the addition result as a to-be-fused text embedding vector;

[0091] Step 4.2, adding the 2D image position embedding, the 1D image position embedding and the masked image information embedding, and taking the addition result as a to-be-fused image embedding vector;

[0092] Step 4.3, connecting the to-be-fused image embedding vector and the to-be-fused text embedding vector to obtain a multi-modal fusion embedding vector;

[0093] The loss calculation module, the pre-training model is trained, the multi-modal fusion embedding vector obtained in step 4.3 is taken as an input of an initial network, different training target weights are performed for different multi-modal document tasks, a word patch alignment target is used to learn fine-grained alignment between text words and image blocks, the word patch alignment target is to predict whether the corresponding image block of the text word is masked, a word patch alignment loss is obtained, and the word patch alignment loss is added to a text loss and an image loss to obtain a total loss of the model;

[0094] The training module, by using the pre-training model, different pre-training models are selected according to different task categories to perform fine-tuning training on different data sets to obtain a fine-tuning model meeting a corresponding task;

[0095] The fine-tuning module, using the fine-tuning model to perform a corresponding document task.

[0096] In the above device, in the extraction module, the 2D text position information is a boundary box coordinate of a text sequence, and the 1D text position information is line number and position offset information in an image block where the text is located.

[0097] The document text content obtains a text information embedding through a trained word table, and the document image information directly serves as an image information embedding.

[0098] The 2D text position information and the 1D text position information obtain a 2D text position embedding and a 1D text position embedding through a position encoding function, wherein the position encoding function can be defined by a user himself or a commonly used position encoding function publicly available on the Internet.

[0099] In the above extraction module,

[0100] The 2D image position information is a boundary box coordinate of an image block.

[0101] 1D image position information is the left-to-right, top-to-bottom sequential numbering of the image block, where the initial number is 1;

[0102] 2D image position information and 1D image position information are obtained by a position encoding function to obtain 2D image position embedding and 1D image position embedding.

[0103] In the above device, different training target weights are performed to perform different pre-training for the text-centered document intelligent task and the image-centered document intelligent task, wherein:

[0104] The training target of the text-centered document intelligent task is a masked sequence context representation based on image embedding and text embedding, and the model maximizes the probability of correctly predicting the text label, and finally obtains the loss of the text-centered pre-training model;

[0105] The image-centered document intelligent task is driven by cross-entropy loss to reconstruct the masked image token x in the context of its surrounding text and image tokens, and the label of the image token x comes from an image tokenizer, which can convert dense image pixels into discrete tokens according to the visual vocabulary, and finally obtain the loss of the image-centered pre-training model.

[0106] The present technical proposal has the following features:

[0107] Although the method of LayoutLMv3 is a good technology in cross-modal document pre-training models, it has some obvious shortcomings, and these shortcomings are the problems solved by the present proposal.

[0108] Firstly, the present proposal proposes a cross-modal document pre-training model that integrates text information and image information, which can better intelligently process document information.

[0109] Secondly, although the parameter amount of the present proposal is very close to that of LayOutLMv3 in the pre-training stage, the present proposal can obtain better results by modifying only a small part of the parameters during the downstream task fine-tuning. In addition, the present proposal also uses the distillation technology to perform model compression, so that the model inference stage is more in line with the speed of real life.

[0110] Finally, this proposal pre-trains the tasks of different modal centers respectively in the pre-training stage, which is text-centered and image-centered, so as to better match the downstream tasks. At the same time, this is also the biggest difference from LayoutLMv3. Because LayoutLMv3 has only one training method for two different tasks, it may perform the financial report analysis task when the model is performing the real-time correction word task, which will affect different tasks and cause the ongoing task to perform poorly.

Claims

1. A method for implementing a document pre-training model that supports cross-modal communication, characterized in that, Includes the following steps: Step 1: Obtain pre-trained document image data; Step 2: The document image is identified as document text information using character recognition methods, obtaining the document text content and corresponding 2D text position information, as well as 1D text position information. At the same time, the 2D image position information and 1D image position information are also obtained. Step 3: Adjust the document image to H × W and represent the image with dimensions C × H × W, where C, H and W are the channel size, width and height of the image, respectively. Then, split the image into a series of uniform P × P blocks, linearly project the image blocks to the dimension and flatten them into vectors to obtain the embedded representation of the image information. Step 4: First, when embedding text information and image information, it is necessary to perform random masking on these two types of embedding vectors to obtain the masked text information embedding and the masked image information embedding. Step 4.1: Add the 2D text position embedding, 1D text position embedding, and masked text information embedding together, and use the result as the text embedding vector to be fused. Step 4.2: Add the 2D image position embedding, 1D image position embedding, and masked image information embedding together, and use the result as the image embedding vector to be fused. Step 4.3: Concatenate the embedding vectors of the image to be fused and the embedding vectors of the text to be fused to obtain the multimodal fusion embedding vector; Step 5: Pre-trained model training. The multimodal fusion embedding vector obtained in step 4.3 is used as the input of the initial network. Different training target weights are applied for multimodal document tasks with different focuses. The word patch alignment target is used to learn fine-grained alignment between text words and image blocks. The word patch alignment target predicts whether the corresponding image block of the text word is masked. The word patch alignment loss is obtained and added to the text loss and image loss respectively to obtain the total model loss. Step 6: Using pre-trained models, select different pre-trained models according to different task categories and fine-tune them on different datasets to obtain fine-tuned models that meet the corresponding tasks. Step 7: Use the fine-tuned model to perform the corresponding document task; Different training objective weights are applied to perform different pre-training processes for text-centric and image-centric document intelligence tasks, respectively. The training objective of the text-centered document intelligence task is to maximize the probability of correctly predicting text tags based on the masked sequence context representation of image embedding and text embedding, and finally obtain the loss of the text-centered pre-trained model. The image-centric document intelligence task is driven by cross-entropy loss to reconstruct a masked image token x in the context of its surrounding text and image tokens. The labels of the image token x come from an image tokenizer that can convert dense image pixels into discrete tokens based on visual vocabulary. Finally, the loss of the image-centric pre-trained model is obtained.

2. The method for implementing a document pre-training model supporting cross-modal communication according to claim 1, characterized in that, In step 2, the 2D text position information refers to the bounding box coordinates of the text sequence, while the 1D text position information refers to the row number and position offset information of the image block in which the text is located. The document text content is embedded using a pre-trained vocabulary, while the document image information is directly embedded as image information. 2D and 1D text location information are used to obtain 2D and 1D text location embeddings through location encoding functions. The location encoding functions can be defined by the user or commonly used location encoding functions publicly available on the Internet.

3. The method for implementing a document pre-training model supporting cross-modal communication according to claim 1, characterized in that, In step 2, The 2D image location information is the bounding box coordinates of the image patch; The 1D image position information consists of the sequential numbering of the image blocks from left to right and from top to bottom, with the initial number being 1; The 2D image location information and the 1D image location information are used to obtain the 2D image location embedding and the 1D image location embedding through the location encoding function.

4. A device for implementing a document pre-training model that supports cross-modal communication, characterized in that, Includes the following modules: Data acquisition module: Obtains pre-trained document image data; Extraction module: The document image is identified as document text information through character recognition method, and the document text content and corresponding 2D text position information and 1D text position information are obtained. At the same time, the 2D image position information and 1D image position information are also obtained. Embedding module: Adjust the document image to H × W and represent the image in the dimension of C × H × W, where C, H and W are the channel size, width and height of the image, respectively. Then, the image is divided into a series of uniform P × P blocks. The image blocks are linearly projected onto the dimension and flattened into vectors, thus obtaining the embedded representation of the image information. Multimodal fusion module: First, when embedding text and image information, the two embedding vectors need to be randomly masked to obtain the masked text and image embeddings, specifically including: The 2D text position embedding, 1D text position embedding, and masked text information embedding are added together, and the result is used as the text embedding vector to be fused. The 2D image location embedding, 1D image location embedding, and masked image information embedding are added together, and the result of the addition is used as the embedding vector of the image to be fused. The multimodal fusion embedding vector is obtained by concatenating the embedding vectors of the image to be fused and the text to be fused. Loss calculation module: The pre-trained model is trained, and the obtained multimodal fusion embedding vector is used as the input of the initial network. Different training target weights are applied for multimodal document tasks with different focuses. The word patch alignment target is used to learn fine-grained alignment between text words and image blocks. The word patch alignment target predicts whether the corresponding image block of the text word is masked, and the word patch alignment loss is obtained. It is then added to the text loss and image loss respectively to obtain the total model loss. The training module uses pre-trained models to select different pre-trained models for different task categories and fine-tunes them on different datasets to obtain fine-tuned models that meet the corresponding tasks. Fine-tuning modules and using fine-tuning models for corresponding document tasks; Different training objective weights are applied to perform different pre-training processes for text-centric and image-centric document intelligence tasks, respectively. The training objective of the text-centered document intelligence task is to maximize the probability of correctly predicting text tags based on the masked sequence context representation of image embedding and text embedding, and finally obtain the loss of the text-centered pre-trained model. The image-centric document intelligence task is driven by cross-entropy loss to reconstruct a masked image token x in the context of its surrounding text and image tokens. The labels of the image token x come from an image tokenizer that can convert dense image pixels into discrete tokens based on visual vocabulary. Finally, the loss of the image-centric pre-trained model is obtained.

5. The device for implementing a document pre-training model supporting cross-modality according to claim 4, characterized in that, In the extraction module, 2D text location information refers to the bounding box coordinates of the text sequence, while 1D text location information refers to the row number and position offset information of the image block in which the text is located. The document text content is embedded using a pre-trained vocabulary, while the document image information is directly embedded as image information. 2D and 1D text location information are used to obtain 2D and 1D text location embeddings through location encoding functions. The location encoding functions can be defined by the user or commonly used location encoding functions publicly available on the Internet.

6. The device for implementing a cross-modal document pre-training model according to claim 4, characterized in that, In the extraction module, The 2D image location information is the bounding box coordinates of the image patch; The 1D image position information consists of the sequential numbering of the image blocks from left to right and from top to bottom, with the initial number being 1; The 2D image location information and the 1D image location information are used to obtain the 2D image location embedding and the 1D image location embedding through the location encoding function.

Images