Certificate identification method and system based on multi-modal model

Abstract

The application discloses a license identification method and system based on a multi-modal model, belongs to the technical field of multi-modal models and image identification, and aims to solve the technical problem of how to directly map input text images to expected structured outputs and improve the accuracy of license identification. The technical scheme is as follows: a multi-modal model based on images and texts is constructed; an OCR model is distilled; a document reading task is pre-trained; license identification data sets are collected; the multi-modal model is fine-tuned; and data post-processing is performed. The system comprises a construction module, a distillation module, a pre-training module, a collection module, a fine-tuning module and a data processing module.

Description

Technical Field

[0001] This invention relates to the field of multimodal modeling and image recognition technology, specifically to a method and system for document recognition based on a multimodal model. Background Technology

[0002] With the continuous development of deep learning technologies, neural network models have been widely applied in many industries and scenarios, especially in general fields such as document recognition. Due to the large amount of data that can be trained on and the greater complexity of the models, neural network models often achieve better results. Deep learning models have become mainstream in the field of document recognition, boasting high accuracy. Document recognition technology is currently widely used in smart cities, intelligent approval processes, and other fields. First, an existing OCR engine is used to read the text, and then the target fields are extracted from the obtained text. The main drawback of these methods is their reliance on an external OCR engine.

[0003] Therefore, how to directly map the input text image to the desired structured output and improve the accuracy of certificate recognition is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] The technical objective of this invention is to provide a document recognition method and system based on a multimodal model to solve the problem of how to directly map the input text image to the desired structured output and improve the accuracy of document recognition.

[0005] The technical objective of this invention is achieved as follows: a certificate recognition method based on a multimodal model, the specific method of which is as follows:

[0006] Constructing multimodal models based on images and text;

[0007] Distillation OCR model;

[0008] Pre-trained document reading task;

[0009] Collect a dataset for document recognition;

[0010] Fine-tuning the multimodal model;

[0011] Data post-processing.

[0012] As a preferred approach, the construction of a multimodal model based on images and text is as follows:

[0013] The multimodal model consists of a Transformer-based visual encoder and a text decoder module; the visual encoder comprises a ConvNet convolutional neural network designed to extract local patterns within characters and a Swin Transformer model designed to capture long-term dependencies; as detailed below:

[0014] The input image is encoded using a visual encoder to generate feature data containing relevant visual information;

[0015] Image feature data and task tokens are fed into the text decoder to automatically regressively generate target tokens; the text decoder consists of n Transformer layers, each consisting of a multi-head self-attention sub-layer, a multi-head cross-attention sub-layer, and a feedforward sub-layer.

[0016] More preferably, the distillation OCR model is as follows:

[0017] The PP-OCR-V3 recognition network is used to apply pointwise convolution to the output based on ConvNet blocks, obtaining the same number of output channels as the PP-OCR-V3 recognition backbone. The loss of the two network outputs is calculated using L2 loss. The ConvNet learns character recognition capabilities through model training and optimization. The training data uses open-source OCR-related datasets.

[0018] More specifically, the pre-trained document reading task is as follows:

[0019] The multimodal model learns to predict the next character based on previous text tags and the input image; the input image is divided into 32×32 blocks, and about 15% of the blocks are masked, allowing the multimodal model to predict the text in the masked blocks;

[0020] The specific data collection for document recognition is as follows:

[0021] Collect relevant datasets for certificate recognition through the internet or business channels, and use commercial OCR software in conjunction with manual annotation of the datasets.

[0022] More specifically, the fine-tuning of the multimodal model is as follows:

[0023] After the pre-training phase, the multimodal model will be fine-tuned for the information extraction task;

[0024] Fine-tuning of the multimodal model on a certificate recognition dataset;

[0025] The feature representation vectors of the hidden state of all output sequences output by the last transformer block of the text decoder are then input into an additional linear multi-classifier for classification, thus completing the block sequence classification.

[0026] More preferably, the data post-processing is as follows:

[0027] Convert the output token sequence to JSON format and add two special markers [Start*] and [End*]; where * represents each field to be extracted;

[0028] If the output token sequence is structurally incorrect, simply treat that field as missing.

[0029] A certificate recognition system based on a multimodal model, the system comprising:

[0030] Builder modules are used to build multimodal models based on images and text;

[0031] The distillation module is used to distill OCR models;

[0032] The pre-training module is used to pre-train document reading tasks;

[0033] The collection module is used to collect certificate recognition datasets;

[0034] The fine-tuning module is used to fine-tune the multimodal model;

[0035] The data processing module is used for post-processing of data.

[0036] Preferably, the multimodal model in the construction module consists of a Transformer-based visual encoder and a text decoder module; wherein, the visual encoder consists of a ConvNet convolutional neural network designed to extract local patterns within characters and a Swin Transformer model designed to capture long-term dependencies; the working process of the construction module is as follows: the input image is encoded by the visual encoder to generate feature data containing relevant visual information; then, the image feature data and the task token are fed into the text decoder to automatically regressively generate the target token; wherein, the text decoder consists of n Transformer layers, each layer consisting of a multi-head self-attention sublayer, a multi-head cross-attention sublayer, and a feedforward sublayer;

[0037] The distillation module works as follows: pointwise convolution is applied to the output based on ConvNet blocks using the PP-OCR-V3 recognition network to obtain the same number of output channels as the PP-OCR-V3 recognition backbone. The loss of the two network outputs is calculated using L2 loss. The ConvNet learns character recognition capabilities through model training and optimization. The training data uses open-source OCR-related datasets.

[0038] The pre-training module works as follows: the multimodal model learns to predict the next character based on previous text tags and the input image; the input image is divided into 32×32 blocks, and about 15% of the blocks are masked, allowing the multimodal model to predict the text in the masked blocks.

[0039] The specific working process of the collection module is as follows: collect relevant datasets for certificate recognition through the Internet or business channels, and use commercial OCR software in conjunction with manual annotation of the datasets;

[0040] The fine-tuning module works as follows: after the pre-training stage, the multimodal model will fine-tune the information extraction task; the multimodal model will be fine-tuned on the certificate recognition dataset; the feature representation vectors of the hidden layer states of all output sequences output by the last layer transformer block of the text decoder will be input into an additional linear multi-classifier for classification, thus completing the block sequence classification.

[0041] The data processing module works as follows: it converts the output token sequence into JSON format and adds two special markers, [Start*] and [End*]. Here, * represents each field to be extracted. If the structure of the output token sequence is incorrect, the field is simply considered to be missing.

[0042] An electronic device includes: a memory and at least one processor;

[0043] The memory contains computer programs;

[0044] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the document recognition method based on the multimodal model as described above.

[0045] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the multimodal model-based document recognition method described above.

[0046] The document recognition method and system based on a multimodal model of the present invention have the following advantages:

[0047] (i) This invention directly maps the input document image to the desired structured output without relying on OCR, and can be easily trained in an end-to-end manner; at the same time, it is different from previous end-to-end methods in that it can better extract distinguishable character features, enabling it to efficiently perceive the internal patterns of characters and the dependencies between characters.

[0048] (II) By combining a multimodal pre-trained model with fine-tuning based on specific task data, the accuracy of the model is greatly improved, which has high practical and innovative value.

[0049] (III) This invention applies the Transformer model to text, images and other fields, and makes it possible to use the end-to-end Transformer model to solve the problem of extracting key information from images. Attached Figure Description

[0050] The invention will be further described below with reference to the accompanying drawings.

[0051] Appendix Figure 1 This is a flowchart of a certificate recognition method based on a multimodal model. Detailed Implementation

[0052] The document recognition method and system based on a multimodal model of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] Example 1:

[0054] As attached Figure 1 As shown in the figure, this embodiment provides a certificate recognition method based on a multimodal model, which is as follows:

[0055] S1. Construct a multimodal model based on images and text;

[0056] S2, Distillation OCR Model;

[0057] S3, Pre-trained document reading task;

[0058] S4. Collect the certificate recognition dataset;

[0059] S5, fine-tuning the multimodal model;

[0060] S6. Data post-processing.

[0061] The construction of the image- and text-based multimodal model in step S1 of this embodiment is as follows:

[0062] The multimodal model consists of a Transformer-based visual encoder and a text decoder module; the visual encoder comprises a ConvNet convolutional neural network designed to extract local patterns within characters and a Swin Transformer model designed to capture long-term dependencies; as detailed below:

[0063] S101. Encode the input image using a visual encoder to generate feature data containing relevant visual information;

[0064] S102. The image feature data and the task token are fed into the text decoder to automatically regressively generate the target token; wherein, the text decoder consists of n Transformer layers, each of which consists of a multi-head self-attention sub-layer, a multi-head cross-attention sub-layer and a feedforward sub-layer.

[0065] The distillation OCR model in step S2 of this embodiment is as follows:

[0066] The PP-OCR-V3 recognition network is used to apply pointwise convolution to the output based on ConvNet blocks, obtaining the same number of output channels as the PP-OCR-V3 recognition backbone. The loss of the two network outputs is calculated using L2 loss. The ConvNet learns character recognition capabilities through model training and optimization. The training data uses open-source OCR-related datasets.

[0067] The pre-trained document reading task in step S3 of this embodiment is as follows:

[0068] The multimodal model learns to predict the next character based on previous text tags and the input image; the input image is divided into 32×32 blocks, and about 15% of the blocks are masked, allowing the multimodal model to predict the text in the masked blocks.

[0069] The specific steps for collecting the certificate recognition dataset in step S4 of this embodiment are as follows:

[0070] Collect relevant datasets for certificate recognition through the internet or business channels, and use commercial OCR software in conjunction with manual annotation of the datasets.

[0071] The fine-tuning of the multimodal model in step S5 of this embodiment is as follows:

[0072] S501. After the pre-training phase, the multimodal model will be fine-tuned for the information extraction task;

[0073] S502. Fine-tune the multimodal model on the certificate recognition dataset;

[0074] S503. The feature representation vectors of the hidden state of all output sequences output by the last transformer block of the text decoder are then input into an additional linear multi-classifier for classification, thus completing the block sequence classification.

[0075] The data post-processing in step S6 of this embodiment is as follows:

[0076] Convert the output token sequence to JSON format and add two special markers [Start*] and [End*]; where * represents each field to be extracted;

[0077] If the output token sequence is structurally incorrect, simply treat that field as missing.

[0078] Example 2:

[0079] This embodiment provides a certificate recognition system based on a multimodal model, the system comprising:

[0080] Builder modules are used to build multimodal models based on images and text;

[0081] The distillation module is used to distill OCR models;

[0082] The pre-training module is used to pre-train document reading tasks;

[0083] The collection module is used to collect certificate recognition datasets;

[0084] The fine-tuning module is used to fine-tune the multimodal model;

[0085] The data processing module is used for post-processing of data.

[0086] The multimodal model in the construction module of this embodiment consists of a Transformer-based visual encoder and a text decoder module. The visual encoder is composed of a ConvNet convolutional neural network designed to extract local patterns within characters and a Swin Transformer model designed to capture long-term dependencies. The construction module works as follows: the input image is encoded by the visual encoder to generate feature data containing relevant visual information; the image feature data and the task token are then fed into the text decoder to automatically regressively generate the target token. The text decoder consists of n Transformer layers, each consisting of a multi-head self-attention sublayer, a multi-head cross-attention sublayer, and a feedforward sublayer.

[0087] The distillation module in this embodiment works as follows: pointwise convolution is applied to the output based on ConvNet blocks using the PP-OCR-V3 recognition network to obtain the same number of output channels as the PP-OCR-V3 recognition backbone, and the loss of the two network outputs is calculated using L2 loss. The ConvNet learns character recognition capabilities through model training and optimization. The training data uses open-source OCR-related datasets.

[0088] The pre-training module in this embodiment works as follows: the multimodal model learns to predict the next character based on the previous text tags and the input image; the input image is divided into 32×32 blocks, and about 15% of the blocks are masked, allowing the multimodal model to predict the text in the masked blocks.

[0089] The collection module in this embodiment works as follows: it collects relevant datasets for certificate recognition through the Internet or business channels, and uses commercial OCR software in conjunction with manual annotation of the datasets.

[0090] The fine-tuning module in this embodiment works as follows: after the pre-training stage, the multimodal model will fine-tune the information extraction task; the multimodal model will be fine-tuned on the certificate recognition dataset; the feature representation vectors of the hidden layer states of all output sequences output by the last transformer block of the text decoder will be input into an additional linear multi-classifier for classification, thus completing the block sequence classification.

[0091] The data processing module in this embodiment works as follows: the output token sequence is converted into JSON format, and two special markers [Start*] and [End*] are added; where * represents each field to be extracted; if the structure of the output token sequence is incorrect, the field is simply considered to be missing.

[0092] Example 3:

[0093] This embodiment also provides an electronic device, including: a memory and a processor;

[0094] The memory stores the instructions executed by the computer.

[0095] The processor executes computer execution instructions stored in the memory, causing the processor to perform the certificate recognition method based on a multimodal model in any embodiment of the present invention.

[0096] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0097] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.

[0098] Example 4:

[0099] This embodiment also provides a computer-readable storage medium storing multiple instructions, which are loaded by a processor to cause the processor to execute the document recognition method based on a multimodal model according to any embodiment of the present invention. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0100] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0101] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0102] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0103] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A certificate recognition method based on a multimodal model, characterized in that, The method is as follows: Constructing multimodal models based on images and text; Distillation OCR model; Pre-trained document reading task; Collect a dataset for document recognition; Fine-tuning the multimodal model; Data post-processing; The specific steps for constructing a multimodal model based on images and text are as follows: The multimodal model consists of a Transformer-based visual encoder and a text decoder module; the visual encoder comprises a ConvNet convolutional neural network designed to extract local patterns within characters and a Swin Transformer model designed to capture long-term dependencies; as detailed below: The input image is encoded using a visual encoder to generate feature data containing relevant visual information; Image feature data and task tokens are fed into the text decoder to automatically regressively generate target tokens; the text decoder consists of n Transformer layers, each consisting of a multi-head self-attention sub-layer, a multi-head cross-attention sub-layer, and a feedforward sub-layer; The distillation OCR model is as follows: The PP-OCR-V3 recognition network applies pointwise convolution to the ConvNet block-based output to obtain the same number of output channels as the PP-OCR-V3 recognition backbone. The loss of the two network outputs is calculated using L2 loss. The ConvNet learns character recognition capabilities through model training and optimization. The training data uses open-source OCR-related datasets. The pre-trained document reading task is as follows: The multimodal model learns to predict the next character based on previous text tags and the input image; the input image is divided into 32×32 blocks, and 15% of the blocks are masked, allowing the multimodal model to predict the text in the masked blocks; The fine-tuning of the multimodal model is as follows: After the pre-training phase, the multimodal model will be fine-tuned for the information extraction task; Fine-tuning of the multimodal model on a certificate recognition dataset; The feature representation vectors of the hidden state of all output sequences output by the last transformer block of the text decoder are then input into an additional linear multi-classifier for classification, thus completing the block sequence classification. The specific post-processing of the data is as follows: Convert the output token sequence to JSON format and add two special markers [Start∗] and [End∗]; where ∗ represents each field to be extracted; If the structure of the output token sequence is incorrect, this field is considered missing.

2. The certificate recognition method based on a multimodal model according to claim 1, characterized in that, The specific data collection for document recognition is as follows: Collect relevant datasets for certificate recognition through the internet or business channels, and use commercial OCR software in conjunction with manual annotation of the datasets.

3. A certificate recognition system based on a multimodal model, characterized in that, The system includes: Builder modules are used to build multimodal models based on images and text; The distillation module is used to distill OCR models; The pre-training module is used to pre-train document reading tasks; The collection module is used to collect certificate recognition datasets; The fine-tuning module is used to fine-tune the multimodal model; The data processing module is used for data post-processing. The multimodal model in the construction module consists of a Transformer-based visual encoder and a text decoder module. The visual encoder is composed of a ConvNet convolutional neural network designed to extract local patterns within characters and a Swin Transformer model designed to capture long-term dependencies. The construction module works as follows: the input image is encoded by the visual encoder to generate feature data containing relevant visual information; the image feature data and the task token are then fed into the text decoder to automatically regressively generate the target token. The text decoder consists of n Transformer layers, each consisting of a multi-head self-attention sublayer, a multi-head cross-attention sublayer, and a feedforward sublayer. The distillation module works as follows: pointwise convolution is applied to the output based on ConvNet blocks using the PP-OCR-V3 recognition network to obtain the same number of output channels as the PP-OCR-V3 recognition backbone. The loss of the two network outputs is calculated using L2 loss. The ConvNet learns character recognition capabilities through model training and optimization. The training data uses open-source OCR-related datasets. The pre-training module works as follows: the multimodal model learns to predict the next character based on the previous text tags and the input image; the input image is divided into 32×32 blocks, and 15% of the blocks are masked, allowing the multimodal model to predict the text in the masked blocks. The fine-tuning module works as follows: after the pre-training stage, the multimodal model will fine-tune the information extraction task; the multimodal model will be fine-tuned on the certificate recognition dataset; the feature representation vectors of the hidden layer states of all output sequences output by the last layer transformer block of the text decoder will be input into an additional linear multi-classifier for classification, thus completing the block sequence classification. The data processing module works as follows: it converts the output token sequence into JSON format and adds two special markers, [Start∗] and [End∗]; where ∗ represents each field to be extracted; if the structure of the output token sequence is incorrect, the field is considered to be missing.

4. The certificate recognition system based on a multimodal model according to claim 3, characterized in that, The collection module works by collecting relevant datasets for certificate recognition through the Internet or business channels, and using commercial OCR software in conjunction with manual annotation of the datasets.

5. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the document recognition method based on a multimodal model as described in claim 1 or 2.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the multimodal model-based certificate recognition method as described in claim 1 or 2.

Images