Key information extraction method and system based on multi-modal model

Abstract

The application discloses a key information extraction method and system based on a multi-modal model, belongs to the technical field of multi-modal models and information extraction, and aims to solve the technical problem of how to map input document images to expected structured outputs and improve model accuracy. The technical scheme adopted is as follows: a multi-modal model based on images and texts is constructed; a pre-training document image dataset is constructed; the multi-modal model is pre-trained; a key information extraction dataset is constructed; the multi-modal model is fine-tuned; and data post-processing is performed. The system comprises a construction module one, a construction module two, a pre-training module, a construction module three, a fine-tuning module and a data processing module.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-modal model and information extraction, in particular to a key information extraction method and system based on multi-modal model. BACKGROUND

[0002] With the continuous development of intelligence in daily life and work, people's demand and dependence on intelligent office are more intense, and the extraction of key information in documents or even pictures is a key step to realize automatic office. For document key information extraction, traditional rule-based methods cannot meet the actual application requirements, and deep learning methods have become the current mainstream. For the extraction of key information in image documents, most current methods first extract text information through an optical character recognition (OCR) engine, then serialize the OCR output text, and then extract information through natural language processing methods. However, this method includes multiple processing stages, and each stage requires an independent model.

[0003] Therefore, how to map the input document image to the desired structured output and improve the model accuracy is a technical problem to be solved. SUMMARY

[0004] The technical task of the present application is to provide a key information extraction method and system based on a multi-modal model to solve the problem of how to map the input document image to the desired structured output and improve the model accuracy.

[0005] The technical task of the present application is achieved in the following manner, a key information extraction method based on a multi-modal model, which is specifically as follows:

[0006] Constructing a multi-modal model based on images and text;

[0007] Constructing a pre-trained document image dataset;

[0008] Pre-training a multi-modal model;

[0009] Constructing a key information extraction dataset;

[0010] Fine-tuning the multi-modal model;

[0011] Data post-processing.

[0012] As a preferred, the multi-modal model based on images and text is constructed as follows:

[0013] The multi-modal model is composed of a Transformer-based visual encoder and a text decoder module; wherein the visual encoder adopts a Swin Transformer model, and converts an input document image x H×W×C into a set of embedded vectors {zi |z i ∈R d ,1≤i≤n}; where n represents the feature map size or the number of image patches; d represents the dimension of the encoder's latent vectors; the output {z} of the Swing Transformer model is fed into the text decoder; the text decoder uses the BART model, and given {z}, the text decoder generates a token sequence. Among them, y i ∈R v represents the heat vector of the i-th token; v represents the size of the token vocabulary; m represents the hyperparameter, which initializes the decoder model weights using the weights of the pre-trained multilingual BART model.

[0014] More preferably, the pre-trained document image dataset is constructed as follows:

[0015] We collect image data from internet documents and use commercial OCR APIs to build text tags for the collected image data; at the same time, in order to expand the data volume, we use data generation methods to generate a batch of document image data.

[0016] Better yet, the pre-trained multimodal model is a multimodal model pre-trained using a document image dataset;

[0017] During the pre-training phase, text is generated using a unidirectional autoregressive encoding method from top left to bottom right, reducing the mismatch and disconnect between the pre-training phase and the text generation phase. The next word is predicted based on the image and the previous context label, and the model loss is calculated using the cross-entropy loss function.

[0018] More optimally, the key information extraction dataset is constructed as follows:

[0019] Downstream tasks based on key information extraction in a specified domain collect relevant datasets and annotate the data to obtain datasets for key information extraction.

[0020] More specifically, the fine-tuning of the multimodal model is as follows:

[0021] The document image data is input into the visual encoder and text decoder of the multimodal model, and the feature representation vectors of the hidden layer states of all output sequences of the last transformer block of the visual decoder are obtained.

[0022] By inputting the feature representation vectors of the hidden layer states of all output sequences into an additional linear multi-classifier for classification, block sequence classification can be completed.

[0023] After classification, the text decoder is trained to generate a token sequence, which is then converted into JSON format representing the desired output information.

[0024] The data post-processing is specifically as follows:

[0025] The output token sequence is converted into a JSON format, and two special marks [Start*] and [End*] are added; wherein, * represents each field to be extracted;

[0026] If the structure of the output token sequence is wrong, the field is simply regarded as missing.

[0027] A key information extraction system based on a multi-modal model, the system comprising:

[0028] A first construction module for constructing a multi-modal model based on images and texts;

[0029] A second construction module for constructing a pre-training document image dataset;

[0030] A pre-training module for pre-training the multi-modal model;

[0031] A third construction module for constructing a key information extraction dataset;

[0032] A fine-tuning module for fine-tuning the multi-modal model;

[0033] A data processing module for data post-processing.

[0034] As a preferred, the working process of the first construction module is specifically as follows: the multi-modal model is composed of a visual encoder based on a Transformer and a text decoder module; wherein, the visual encoder adopts a Swin Transformer model, and an input document image x∈R H×W×C is converted into a group of embedded vectors {z i |z i ∈R d ,1≤i≤n}; wherein, n represents a feature mapping size or a number of image blocks; d represents a dimension of a latent vector of the encoder; the output {z} of the Swin Transformer model is fed into the text decoder; the text decoder adopts a BART model, and the text decoder generates a token sequence wherein, y i ∈R v represents a hot vector of the i-th token; v represents a size of a token vocabulary; m represents a hyperparameter, and the decoder model weight is initialized using the weight of the pre-trained multi-language BART model;

[0035] The working process of the second construction module is specifically as follows: collecting Internet document image data, and using a commercial OCR API to construct text labels for the collected image data; at the same time, in order to expand the data volume, a batch of document image data is generated using a data generation method;

[0036] The pre-training module is used for pre-training the multi-modal model using a document image data set; in the pre-training stage, the text is generated in a left-up to right-down one-way autoregressive encoding manner, so as to weaken the mismatch degree and fragmentation between the pre-training stage and the text generation stage; the next word is predicted based on the image and the previous context text label, and the model loss is calculated using a cross-entropy loss function;

[0037] The third construction module is used for collecting a corresponding data set based on a specified field key information extraction downstream task, and labeling the data to obtain a data set for key information extraction;

[0038] The working process of the fine-tuning module is specifically as follows: the document image data is input into the visual encoder and the text decoder of the multi-modal model, the feature representation vectors of the hidden layer states of all output sequences output by the last layer of the visual decoder are obtained, and then the feature representation vectors of the hidden layer states of all output sequences are input into an additional linear multi-classifier for classification, so that the block sequence classification is completed; after classification, the text decoder is trained to generate a token sequence, and the token sequence is converted into a JSON format representing the required output information;

[0039] The data processing module is used for converting the output token sequence into a JSON format, and adding two special marks [Start*] and [End*]; wherein, * represents each field to be extracted; if the structure of the output token sequence is incorrect, the field is simply regarded as missing.

[0040] An electronic device, comprising a memory and at least one processor;

[0041] The memory stores a computer program;

[0042] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the key information extraction method based on the multi-modal model as described above.

[0043] A computer readable storage medium, the computer readable storage medium stores a computer program, the computer program can be executed by a processor to realize the key information extraction method based on the multi-modal model as described above.

[0044] The key information extraction method and system based on the multi-modal model have the following advantages:

[0045] (I) The application uses the multi-modal model to extract key information from a document image, and sequentially passes through the steps of constructing a multi-modal model based on images and texts, constructing a pre-training document image dataset, pre-training a multi-modal model, constructing a key information extraction dataset, fine-tuning the multi-modal model, and data post-processing, directly mapping the input document image to the expected structured output, and the model processing efficiency is higher;

[0046] (II) Unlike traditional methods, the application does not rely on OCR and can be easily trained in an end-to-end manner. By combining a multi-modal pre-training model and specific task data fine-tuning, the model accuracy is greatly improved, and has high practical value and innovative value. BRIEF DESCRIPTION OF DRAWINGS

[0047] The application will be further described below with reference to the accompanying drawings.

[0048] ATTACHMENT Figure 1 The flowchart of the key information extraction method based on the multi-modal model. DETAILED DESCRIPTION

[0049] The key information extraction method and system based on the multi-modal model will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] Embodiment 1:

[0051] The embodiment provides a key information extraction method based on a multi-modal model, which specifically comprises the following steps:

[0052] S1, constructing a multi-modal model based on images and texts;

[0053] S2, constructing a pre-training document image dataset;

[0054] S3, pre-training a multi-modal model;

[0055] S4, constructing a key information extraction dataset;

[0056] S5, fine-tuning the multi-modal model;

[0057] S6, data post-processing.

[0058] The multi-modal model based on images and texts in step S1 of the embodiment is constructed as follows:

[0059] The multi-modal model is composed of a visual encoder based on a Transformer and a text decoder module; wherein the visual encoder adopts a Swin Transformer model, and converts the input document image x H×W×C into a set of embedded vectors {z i |z i ∈Rd ,1≤i≤n}; where n represents the feature map size or the number of image patches; d represents the dimension of the encoder's latent vectors; the output {z} of the Swing Transformer model is fed into the text decoder; the text decoder uses the BART model, and given {z}, the text decoder generates a token sequence. Among them, y i ∈R v represents the heat vector of the i-th token; v represents the size of the token vocabulary; m represents the hyperparameter, which initializes the decoder model weights using the weights of the pre-trained multilingual BART model.

[0060] The specific steps for constructing the pre-trained document image dataset in step S2 of this embodiment are as follows:

[0061] We collect image data from internet documents and use commercial OCR APIs to build text tags for the collected image data; at the same time, in order to expand the data volume, we use data generation methods to generate a batch of document image data.

[0062] In this embodiment, the pre-trained multimodal model in step S3 is a multimodal model pre-trained using a document image dataset;

[0063] During the pre-training phase, text is generated using a unidirectional autoregressive encoding method from top left to bottom right, reducing the mismatch and disconnect between the pre-training phase and the text generation phase. The next word is predicted based on the image and the previous context label, and the model loss is calculated using the cross-entropy loss function.

[0064] The specific steps for constructing the key information extraction dataset in step S4 of this embodiment are as follows:

[0065] Downstream tasks based on key information extraction in a specified domain collect relevant datasets and annotate the data to obtain datasets for key information extraction.

[0066] The fine-tuning of the multimodal model in step S5 of this embodiment is as follows:

[0067] S501. Input the document image data into the visual encoder and text decoder of the multimodal model, and obtain the feature representation vector of the hidden layer state of all output sequences of the last transformer block of the visual decoder.

[0068] S502. Input the feature representation vectors of the hidden layer states of all output sequences into an additional linear multi-classifier for classification to complete the block sequence classification.

[0069] S503. After classification, the text decoder is trained to generate a token sequence, which is then converted into JSON format representing the desired output information.

[0070] The data post-processing in step S6 of this embodiment is as follows:

[0071] Convert the output token sequence to JSON format and add two special markers [Start*] and [End*]; where * represents each field to be extracted;

[0072] If the output token sequence is structurally incorrect, simply treat that field as missing.

[0073] Example 2:

[0074] This embodiment provides a key information extraction system based on a multimodal model, the system comprising:

[0075] Module 1 is used to build multimodal models based on images and text;

[0076] Module 2 is used to build the pre-trained document image dataset;

[0077] The pre-training module is used to pre-train multimodal models;

[0078] Module 3 is used to build a dataset for extracting key information;

[0079] The fine-tuning module is used to fine-tune the multimodal model;

[0080] The data processing module is used for post-processing of data.

[0081] The working process of the first construction module in this embodiment is as follows: the multimodal model consists of a Transformer-based visual encoder and a text decoder module; wherein, the visual encoder adopts the Swing Transformer model, which converts the input document image x∈R... H×W×C Convert into a set of embedded vectors {z i |z i ∈R d ,1≤i≤n}; where n represents the feature map size or the number of image patches; d represents the dimension of the encoder's latent vectors; the output {z} of the Swing Transformer model is fed into the text decoder; the text decoder uses the BART model, and given {z}, the text decoder generates a token sequence. Among them, y i ∈R v represents the heat vector of the i-th token; v represents the size of the token vocabulary; m represents the hyperparameter, which initializes the decoder model weights using the weights of the pre-trained multilingual BART model.

[0082] The working process of the second construction module in this embodiment is as follows: collect Internet document image data and use the API of commercial OCR to construct text tags for the collected image data; at the same time, in order to expand the data volume, use data generation methods to generate a batch of document image data.

[0083] In this embodiment, the pre-training module uses a document image dataset to pre-train a multimodal model. During the pre-training phase, text is generated using a unidirectional autoregressive encoding method from top left to bottom right to reduce the mismatch and disconnect between the pre-training phase and the text generation phase. The next word is predicted based on the image and the previous context label, and the model loss is calculated using the cross-entropy loss function.

[0084] In this embodiment, the third construction module collects the corresponding dataset based on the downstream task of extracting key information in a specified domain, and annotates the data to obtain a dataset for key information extraction.

[0085] The fine-tuning module in this embodiment works as follows: document image data is input into the visual encoder and text decoder of the multimodal model to obtain the feature representation vectors of the hidden layer states of all output sequences output by the last transformer block of the visual decoder; then the feature representation vectors of the hidden layer states of all output sequences are input into an additional linear multi-classifier for classification to complete the block sequence classification; after classification, the text decoder is trained to generate a token sequence, which is converted into JSON format representing the required output information.

[0086] In this embodiment, the data processing module converts the output token sequence into JSON format and adds two special markers, [Start*] and [End*]. Here, * represents each field to be extracted; if the structure of the output token sequence is incorrect, the field is simply considered to be missing.

[0087] Example 3:

[0088] This embodiment also provides an electronic device, including: a memory and a processor;

[0089] The memory stores the instructions executed by the computer.

[0090] The processor executes computer execution instructions stored in the memory, causing the processor to execute the key information extraction method based on a multimodal model in any embodiment of the present invention.

[0091] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0092] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.

[0093] Example 4:

[0094] This embodiment also provides a computer-readable storage medium storing multiple instructions, which are loaded by a processor to cause the processor to execute the key information extraction method based on a multimodal model according to any embodiment of the present invention. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0095] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0096] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0097] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0098] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for key information extraction based on a multi-modal model, characterized in that, The method specifically comprises the following steps: constructing a multi-modal model based on images and texts; constructing a pre-training document image dataset; pre-training the multi-modal model; constructing a key information extraction dataset; fine-tuning the multi-modal model; data post-processing; The multi-modal model based on images and texts is constructed specifically as follows: The multi-modal model is composed of a Transformer-based visual encoder and a text decoder module; wherein the visual encoder adopts a Swin Transformer model, converts the input document image x e R H×W×C into a set of embedded vectors {z i |z i i e R d , 1≤i≤n}; wherein n represents the feature mapping size or the number of image blocks; d represents the dimension of the latent vector of the encoder; the output {z} of the Swin Transformer model is fed into the text decoder; the text decoder adopts a BART model, and the text decoder generates a token sequence wherein y i i e R v represents the hot vector of the i-th token; v represents the size of the token vocabulary; m represents a hyperparameter, and the decoder model weight is initialized using the weight of the pre-trained multi-language BART model; The multi-modal model is fine-tuned specifically as follows: The document image data is input into the visual encoder and the text decoder of the multi-modal model, the feature representation vectors of the hidden layer states of all output sequences output by the last layer of the transformer block of the visual decoder are obtained, and the feature representation vectors are input into an additional linear multi-classifier for classification, so that the block sequence classification is completed. After classification, the text decoder is trained to generate a token sequence, and the token sequence is converted into a JSON format representing the required output information. The data post-processing is specifically as follows: The output token sequence is converted into a JSON format, and two special marks [Start*] and [End*] are added; wherein, * represents each field to be extracted. If the structure of the output token sequence is incorrect, the field is regarded as missing. The pre-training document image dataset is constructed specifically as follows: 2.The key information extraction method based on a multi-modal model according to claim 1, characterized in that, Internet document image data is collected, and a commercial OCR API is used to construct text labels for the collected image data; at the same time, a batch of document image data is generated using a data generation method. The multi-modal model is pre-trained using the document image dataset. 3.The key information extraction method based on a multi-modal model according to claim 1 or 2, characterized in that, In the pre-training stage, the text is generated in a left-up to right-down one-way autoregressive encoding manner, the mismatch and fragmentation between the pre-training stage and the text generation stage are reduced, the next word is predicted based on the image and the previous context text label, and the model loss is calculated using a cross-entropy loss function. The key information extraction dataset is constructed specifically as follows: 4.The key information extraction method based on a multi-modal model according to claim 3, characterized in that, A corresponding dataset is collected based on the key information extraction downstream task of a specified field, and the data is labeled to obtain a dataset for key information extraction. The system comprises: 5.A key information extraction system based on a multi-modal model, characterized by, a first construction module for constructing a multi-modal model based on images and texts; a second construction module for constructing a pre-training document image dataset; a pre-training module for pre-training the multi-modal model; a third construction module for constructing a key information extraction dataset; a fine-tuning module for fine-tuning the multi-modal model; a data processing module for data post-processing; The working process of the fine-tuning module is specifically as follows: document image data is input into the visual encoder and the text decoder of the multi-modal model, the feature representation vectors of the hidden layer states of all output sequences output by the last layer of the transformer block of the visual decoder are obtained, and the feature representation vectors are input into an additional linear multi-classifier for classification, so that the block sequence classification is completed. Wherein, the working process of the construction module one is specifically: the multi-modal model is composed of a visual encoder based on Swin Transformer model and a text decoder module; wherein, the visual encoder adopts the Swin Transformer model, and the input document image x∈R H×W×C is converted into a set of embedded vectors {z i |z i ∈R d ,1≤i≤n}; wherein, n represents the feature mapping size or the number of image blocks; d represents the dimension of the latent vector of the encoder; the output {z} of the Swin Transformer model is fed into the text decoder; the text decoder adopts the BART model, and the text decoder generates a token sequence wherein, y i ∈R v represents the hot vector of the i th token; v represents the size of the token vocabulary; m represents a hyperparameter, and the decoder model weight is initialized using the weight of the pre-trained multilingual BART model; After classification, the text decoder is trained to generate a token sequence, and the token sequence is converted into a JSON format representing the required output information. The data processing module is to convert the output token sequence into JSON format, and add two special marks [Start*] and [End*]; wherein, * represents each field to be extracted; if the structure of the output token sequence is wrong, the field is regarded as missing. 6.The key information extraction system based on a multi-modal model according to claim 5, characterized in that, The working process of the second building module is specifically: collecting Internet document image data, and using a commercial OCR API to build text labels for the collected image data; at the same time, a batch of document image data is generated using a data generation method; The pre-training module is to pre-train the multi-modal model using a document image data set; wherein, in the pre-training stage, the text is generated in a left-to-right and top-to-bottom one-way autoregressive encoding manner to reduce the mismatch and fragmentation between the pre-training stage and the text generation stage; and the next word is predicted based on the image and the previous context text label, and the model loss is calculated using a cross-entropy loss function; The third building module is to collect a corresponding data set based on the key information extraction downstream task of a specified field, and to label the data to obtain a data set for key information extraction.

7. An electronic device, comprising: It comprises: a memory and at least one processor; wherein the memory has stored thereon a computer program; the at least one processor executes the computer program stored in the memory, so that the at least one processor executes the key information extraction method based on the multi-modal model according to any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium has stored therein a computer program, which can be executed by a processor to implement the key information extraction method based on the multi-modal model according to any one of claims 1 to 4.

Images