Document image information extraction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a document image information extraction method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence and finance. The method comprises the following steps: acquiring an original document image; performing generative information extraction on original document information in the original document image to obtain target information extraction data; performing optical character recognition on original document information in the original document image to obtain target character recognition data; determining a model configuration operation instruction based on the original document image; determining an information fusion reference sample according to the original document image and the model configuration operation instruction; performing model configuration on the original information fusion model according to the model configuration operation instruction and the information fusion reference sample to obtain a target information fusion model; and performing information fusion on the target information extraction data and the target character recognition data through a target information fusion model to obtain target document image information. According to the embodiment of the invention, the accuracy of document image information extraction can be effectively improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and financial technology, and in particular to a document image information extraction method and device, electronic equipment and storage medium. Background Art

[0002] Document images refer to files that are converted into image formats by scanning, photographing, etc. for storage. Document image information extraction refers to the process of extracting the required text, structured data or other information from document images. It can be applied to a variety of scenarios, such as the financial field, to extract information from document images such as financial statements, contracts, and bills.

[0003] At present, machine learning models are mainly used to extract information from document images. However, in actual application scenarios, there are many types of document images, and machine learning models cannot adapt to multiple types of document images, resulting in poor accuracy in document image information extraction.

[0004] Therefore, how to improve the accuracy of document image information extraction has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a document image information extraction method and device, electronic device and storage medium, aiming to improve the accuracy of document image information extraction.

[0006] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a document image information extraction method, the method comprising:

[0007] Acquire an original document image; wherein the original document image contains original document information;

[0008] Performing generative information extraction on the original document information in the original document image to obtain target information extraction data;

[0009] Performing optical character recognition on the original document information in the original document image to obtain target character recognition data;

[0010] determining a model configuration operation instruction based on the original document image;

[0011] Determine an information fusion reference sample according to the original document image and the model configuration operation instruction;

[0012] Performing model configuration on a preset original information fusion model according to the model configuration operation instruction and the information fusion reference sample to obtain a target information fusion model;

[0013] The target information extraction data and the target character recognition data are fused through the target information fusion model to obtain target document image information.

[0014] In some embodiments, determining a model configuration operation instruction based on the original document image includes:

[0015] Performing type recognition on the original document image to obtain a target document type;

[0016] Determine target output slot information according to the target document type;

[0017] Performing defect detection on the target information extraction data to obtain information extraction defect information;

[0018] Performing defect detection on the target character recognition data to obtain character recognition defect information;

[0019] The information extraction defect information, the character recognition defect information and the target output slot information are filled into a preset instruction template to obtain the model configuration operation instruction.

[0020] In some embodiments, determining the information fusion reference sample according to the original document image and the model configuration operation instruction includes:

[0021] Acquiring a document image sample from a preset sample database based on the target document type;

[0022] The information fusion reference sample is generated based on the document image sample and the model configuration operation instruction.

[0023] In some embodiments, the document image sample includes a sample document image and expected document image information, the sample document image contains sample document information, and the expected document image information is used to represent document image information expected to be extracted from the sample document image; generating the information fusion reference sample based on the document image sample and the model configuration operation instruction includes:

[0024] Performing generative information extraction on the sample document information in the sample document image to obtain sample information extraction data;

[0025] Performing optical character recognition on the sample document information in the sample document image to obtain sample character recognition data;

[0026] The sample information extraction data, the sample character recognition data, the model configuration operation instruction and the expected document image information are combined to obtain the information fusion reference sample.

[0027] In some embodiments, performing generative information extraction on the original document information in the original document image to obtain target information extraction data includes:

[0028] Performing feature encoding on the original document image to obtain document image features;

[0029] Feature decoding is performed on the document image features to obtain the target information extraction data.

[0030] In some embodiments, the step of decoding the document image features to obtain the target information extraction data includes:

[0031] Performing attention calculation on the document image features to obtain image text distribution features;

[0032] Performing context calculation on the image text distribution features to obtain original text features;

[0033] Performing text prediction on the original text features to obtain text prediction probability data;

[0034] Determining target text information based on the text prediction probability data;

[0035] The target information extraction data is generated according to the target text information.

[0036] In some embodiments, obtaining the original document image includes:

[0037] Obtaining an initial document image;

[0038] Performing denoising processing on the initial document image to obtain a denoised document image;

[0039] Performing image enhancement processing on the denoised document image to obtain an enhanced document image;

[0040] The enhanced document image is resized to obtain the original document image.

[0041] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a document image information extraction device, the device comprising:

[0042] A document image acquisition module, used to acquire an original document image; wherein the original document image contains original document information;

[0043] A generative information extraction module, used for performing generative information extraction on the original document information in the original document image to obtain target information extraction data;

[0044] An optical character recognition module, used for performing optical character recognition on the original document information in the original document image to obtain target character recognition data;

[0045] An operation instruction determination module, used for determining a model configuration operation instruction based on the original document image;

[0046] A reference sample determination module, used to determine an information fusion reference sample according to the original document image and the model configuration operation instruction;

[0047] A model configuration module, used to configure a preset original information fusion model according to the model configuration operation instruction and the information fusion reference sample to obtain a target information fusion model;

[0048] The information fusion module is used to fuse the target information extraction data and the target character recognition data through the target information fusion model to obtain the target document image information.

[0049] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0050] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0051] The document image information extraction method and device, electronic device and storage medium proposed in the present application obtain an original document image; wherein the original document image contains original document information; generative information extraction is performed on the original document information in the original document image to obtain target information extraction data, and the complex layout and context information of the document can be learned; optical character recognition is performed on the original document information in the original document image to obtain target character recognition data, thereby ensuring accurate recognition of text content; model configuration operation instructions are determined based on the original document image; information fusion reference samples are determined according to the original document image and the model configuration operation instructions; a preset original information fusion model is configured according to the model configuration operation instructions and the information fusion reference samples to obtain a target information fusion model; information fusion is performed on the target information extraction data and the target character recognition data through the target information fusion model to obtain more accurate and complete target document image information, thereby effectively improving the comprehensiveness, accuracy and efficiency of document image information extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1is a flow chart of a method for extracting document image information provided by an embodiment of the present application;

[0053] Figure 2 yes Figure 1 Flow chart of step S101 in FIG.

[0054] Figure 3 yes Figure 1 Flow chart of step S102 in FIG.

[0055] Figure 4 yes Figure 3 Flow chart of step S302 in FIG.

[0056] Figure 5 yes Figure 1 Flow chart of step S104 in FIG.

[0057] Figure 6 yes Figure 1 Flow chart of step S105 in FIG.

[0058] Figure 7 yes Figure 6 Flowchart of step S602 in FIG.

[0059] Figure 8 It is a structural schematic diagram of a document image information extraction device provided in an embodiment of the present application;

[0060] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0064] First, some nouns involved in this application are analyzed:

[0065] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0066] Document images refer to image representations of paper or electronic documents obtained by scanning, photographing or other digital means. Document images retain the original layout, format, text content, and possible visual elements such as graphics and tables of the document, making it easier to store, transmit, process and retrieve in a digital environment. Document images are widely used in many fields such as office automation, information archiving, and data mining, and are an important bridge connecting traditional paper documents with modern information technology.

[0067] Generative model, a generative model is a model-based method that generates outputs that match given conditions or inputs. In probability and statistics theory, a generative model refers to a model that can randomly generate observation data and assign a joint probability distribution to the observation value and the labeled data sequence. Generative models can be deep learning models, neural network models, etc. These models have mastered the distribution characteristics of data by learning from data sets, and thus have the ability to generate new data. In the field of natural language processing, generative models are often used to generate natural language sequences such as texts, articles, and dialogues, such as machine translation, summary generation, and dialogue systems.

[0068] Optical Character Recognition (OCR) is a technology that converts printed or handwritten text in an image into machine-encoded text. By scanning a document image, OCR software can analyze the shape, arrangement, and size of the characters in the image and recognize them as corresponding digitally encoded characters (such as ASCII or Unicode), allowing computers to read, edit, and search text information that originally existed in the form of images. OCR technology is widely used in document digitization, information extraction, automated office, and other fields, greatly improving the efficiency and accuracy of text processing.

[0069] Document image information extraction refers to the process of automatically identifying and extracting the required text, structured data or other information from digitized document images. Document image information extraction technology is widely used in office automation, digital libraries, archive management, automatic form processing and other fields. It can significantly improve the efficiency and accuracy of information processing and achieve rapid retrieval, analysis and reuse of documents. Through document image information extraction, users can easily obtain key information in documents without manual input or copying, which greatly reduces the workload.

[0070] Document image information extraction technology can also be applied in financial scenarios to extract information from document images such as financial statements, contracts, and bills.

[0071] At present, document image information extraction technology is mainly classified into two categories:

[0072] One type is an extractive model based on the OCR results of document images. The performance of this type of extractive model depends largely on the accuracy of OCR recognition. Although existing OCR technology performs well in Chinese recognition, the OCR results may be inaccurate when processing fields containing line breaks or spaces, thus affecting the recognition effect of the sequence labeling model.

[0073] The other type is a generative model based on document images. This type of generative model does not rely on OCR results, but directly generates the content of the extracted field based on the features of the image. Therefore, even if there are line breaks or spaces in the field content, the relevant content can be accurately identified. However, the essence of this type of generative model is a machine learning model / deep learning model, and most of them are trained based on large-scale English corpora and relatively small-scale Chinese corpora, which makes them prone to errors when processing Chinese content. Moreover, in actual application scenarios, there are many types of document images, and the generative model cannot adapt to multiple types of document images, resulting in poor accuracy in document image information extraction.

[0074] Based on this, the embodiments of the present application provide a document image information extraction method and device, an electronic device and a storage medium, aiming to improve the accuracy of document image information extraction.

[0075] The document image information extraction method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the document image information extraction method in the embodiments of the present application is described.

[0076] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0077] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0078] The document image information extraction method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The document image information extraction method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the document image information extraction method, etc., but is not limited to the above forms.

[0079] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0080] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0081] Figure 1 is an optional flowchart of the document image information extraction method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0082] Step S101, obtaining an original document image; wherein the original document image contains original document information;

[0083] Step S102, performing generative information extraction on the original document information in the original document image to obtain target information extraction data;

[0084] Step S103, performing optical character recognition on the original document information in the original document image to obtain target character recognition data;

[0085] Step S104, determining a model configuration operation instruction based on the original document image;

[0086] Step S105, determining an information fusion reference sample according to the original document image and the model configuration operation instruction;

[0087] Step S106, configuring the preset original information fusion model according to the model configuration operation instruction and the information fusion reference sample to obtain a target information fusion model;

[0088] Step S107, performing information fusion on the target information extraction data and the target character recognition data through the target information fusion model to obtain the target document image information.

[0089] Steps S101 to S107 shown in the embodiment of the present application, by obtaining target information extraction data through generative information extraction technology in document image information extraction, can learn the complex layout and context information of the document, obtain target character recognition data through optical character recognition technology, ensure accurate recognition of text content, and dynamically configure the model based on the original document image, and finally use the target information fusion model to fuse the target information extraction data and the target character recognition data, so as to infer more precise and complete target document image information, effectively improving the comprehensiveness, accuracy and efficiency of document image information extraction.

[0090] It should be noted that the original document image contains original document information, wherein the original document information is text information, table information, punctuation information, graphic information, layout information, etc. in the original document image, but is not limited thereto.

[0091] In the financial field, the original document image may be a financial statement, a contract, a bill, etc.

[0092] For example, if the original document image is a receipt, the original document information may include:

[0093] Ticket Number: The ticket's unique number, used for identification and tracking.

[0094] Date: The date on which the instrument is issued or the date on which the transaction is made, which is essential in determining the timing and sequence of transactions.

[0095] Payer and payee information: including name, address, contact information, etc., to clarify the identities of both parties to the transaction.

[0096] Transaction amount: The transaction amount indicated on the bill.

[0097] Transaction type: such as sales, purchases, services, etc., indicating the nature of the transaction.

[0098] Description of goods or services: A brief description of the goods or services involved in the transaction will help understand the specific content of the transaction.

[0099] Payment method: Indicate how the transaction is paid, such as cash, bank transfer, check, etc.

[0100] Signature and Seal: The bill usually has the signature of the relevant person or the seal of the unit to confirm the authenticity and validity of the bill.

[0101] Security markings: Tickets may contain special security markings or watermarks to prevent counterfeiting.

[0102] Remarks: Additional explanation of the transaction or other relevant matters.

[0103] Attachment information: If the bill involves other documents or slips, the number or name of the attachment may be noted here.

[0104] It should be noted that the specific information contained in different types of bills (such as invoices, receipts, checks, etc.) may be different, but is not limited to this.

[0105] See also Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S204:

[0106] Step S201, obtaining an initial document image;

[0107] Step S202, performing denoising processing on the initial document image to obtain a denoised document image;

[0108] Step S203, performing image enhancement processing on the denoised document image to obtain an enhanced document image;

[0109] Step S204: resize the enhanced document image to obtain the original document image.

[0110] In steps S201 to S204 shown in the embodiment of the present application, before document image information is extracted, an initial document image is obtained, and denoising, image enhancement and resizing are performed on the initial document image in sequence to obtain the original document image, thereby improving the image clarity and increasing the compatibility of the document image, which helps to improve the quality and efficiency of subsequent information extraction, thereby inferring more accurate and complete document content information.

[0111] It should be noted that the initial document image is the image to be extracted for information, that is, the most initial image, and therefore needs to undergo a corresponding preprocessing process before being input into the model.

[0112] In step S202 of some embodiments, a filter or a noise reduction algorithm may be used to remove noise in the initial document image, thereby improving the accuracy of subsequent information extraction.

[0113] In step S203 of some embodiments, the quality of the image is enhanced by adjusting contrast, brightness, color balance, etc., so as to further highlight the key information in the document image, so as to improve the accuracy of subsequent processing.

[0114] In step S204 of some embodiments, the image size is scaled to a preset size to meet the model's requirement for the input image size, thereby ensuring the compatibility of the document image with subsequent processing algorithms and avoiding information loss or processing errors due to size mismatch.

[0115] In step S102 of some embodiments, generative information extraction is performed on original document information in the original document image using a preset generative document image extraction model to obtain target information extraction data;

[0116] It should be noted that the generative document image extraction model is a pre-trained model.

[0117] In some embodiments, the generative document image extraction model includes an image encoder and a text decoder. The image encoder extracts image features from the original document image, and the text decoder maps the image features to text features, thereby obtaining target information extraction data.

[0118] See also Figure 3 In some embodiments, step S102 may include but is not limited to steps S301 to S302:

[0119] Step S301, feature encoding the original document image to obtain document image features;

[0120] Step S302, feature decoding the document image features to obtain target information extraction data. Steps S301 to S302 shown in the embodiment of the present application, by feature encoding the original document image to obtain document image features, and then feature decoding the document image features to obtain target information extraction data, can effectively capture and utilize the complex feature information in the document image, improve the accuracy and efficiency of information extraction, and also enable the generative document image extraction model to adapt to different types of document images, realizing a more intelligent and flexible information extraction process.

[0121] In step S301 of some embodiments, feature encoding is performed on the original document image by an image encoder in a generative document image extraction model to obtain document image features.

[0122] Among them, the image encoder can be a neural network model (such as VGG, ResNet and other models), an autoencoder, a variational autoencoder, etc., and the specific selection needs to be based on the actual application scenario, but is not limited to this.

[0123] In step S302 of some embodiments, the document image features are feature decoded by a text decoder in a generative document image extraction model to obtain target information extraction data.

[0124] Among them, the text encoder can be a sequence-to-sequence (Seq2Seq) model architecture, such as an RNN (LSTM, GRU)-based decoder or a Transformer-based decoder. The specific selection needs to be based on the actual application scenario, but is not limited to this.

[0125] See also Figure 4 In some embodiments, step S302 may include but is not limited to steps S401 to S405:

[0126] Step S401, performing attention calculation on document image features to obtain image text distribution features;

[0127] Step S402, performing context calculation on the image text distribution features to obtain original text features;

[0128] Step S403, performing text prediction on the original text features to obtain text prediction probability data;

[0129] Step S404, determining target text information based on the text prediction probability data;

[0130] Step S405: generating target information extraction data according to the target text information.

[0131] In steps S401 to S405 shown in the embodiment of the present application, in the text decoder of the generative document image extraction model, a series of steps such as attention calculation, context calculation and text prediction are performed on the document image features to restore the original text features that are closer to the original text information, which can significantly improve the accuracy and relevance of the target information extraction, and then generate target information extraction data based on the target text information to ensure the accuracy of the target text information.

[0132] In step S401 of some embodiments, an attention mechanism is used to perform attention calculation on document image features to obtain image text distribution features, wherein the image text distribution features are used to indicate which parts of the document image should be paid attention to when generating a certain word / term.

[0133] In step S402 of some embodiments, the original text feature is obtained by performing context calculation on the image text distribution feature, wherein the original text feature is a context vector, and the context vector contains key image information required to generate a certain word / term.

[0134] In step S403 of some embodiments, text prediction is performed on the original text features (context vectors) to obtain text prediction probability data, wherein the text prediction probability data is a probability distribution value of a certain word / term;

[0135] Then, according to the probability distribution value of the word / term, through greedy search or beam search and other algorithms, the word / term with the most matching probability distribution value is selected from the preset text probability distribution database as the predicted text, thereby obtaining the target text information;

[0136] It should be noted that the target text information is specifically individual characters / words / phrases, so the characters / words / phrases need to be converted into complete sentences and texts to obtain target information extraction data.

[0137] In step S103 of some embodiments, optical character recognition is performed on the original document information in the original document image by using a preset optical character recognition model to obtain target character recognition data;

[0138] Among them, the preset optical character recognition model is a pre-trained OCR model, such as a hidden Markov model (HMM), an OCR model based on deep learning (such as CNN, RNN, LSTM, etc.), etc., but is not limited thereto.

[0139] It should be noted that both the preset generative document image extraction model and the preset optical character recognition model have their own defects, so it is necessary to fuse the results obtained from the two to improve the accuracy of document image information extraction.

[0140] Figure 5 In some embodiments, step S104 may also include but is not limited to steps S501 to S505:

[0141] Step S501, performing type recognition on the original document image to obtain the target document type;

[0142] Step S502, determining target output slot information according to the target document type;

[0143] Step S503, performing defect detection on the target information extraction data to obtain information extraction defect information;

[0144] Step S504, performing defect detection on the target character recognition data to obtain character recognition defect information;

[0145] Step S505 , filling the information extraction defect information, the character recognition defect information and the target output slot information into a preset instruction template to obtain a model configuration operation instruction.

[0146] Steps S501 to S505 shown in the embodiment of the present application obtain the target document type by performing type recognition on the original document image, and determine the target output slot information based on the target document type. At the same time, defects in the information extraction and character recognition processes are detected, and this information is integrated into the model configuration operation instructions. This can effectively improve the accuracy and flexibility of information processing and provide targeted guidance for subsequent model configuration and multi-source result description.

[0147] It should be noted that the target document type is used to indicate the document type corresponding to the original document image. For example, in the financial field scenario, the target document type can be a financial statement type, a contract type, a bill type, etc., or even more detailed, such as bills including invoice types, receipt types, etc.

[0148] In some embodiments, the target document type may be preset by a technician, or may be identified by a preset document image type recognition model, but is not limited thereto.

[0149] In step S502 of some embodiments, the target output slot information corresponds to the target document type and is preset by a technician.

[0150] For example, if the target document type of the original document image is a bill type, the target output slot information includes: bill number, date, transaction amount, payer, payee, and transaction method;

[0151] If the target document type of the original document image is a property ownership certificate type, the target output slot information includes: property certificate number, property owner, location, area, and holding time.

[0152] It should be noted that the setting of the target output slot information needs to be set according to the actual application scenario, and is not limited thereto.

[0153] In step S503 of some embodiments, the target information extraction data is obtained by processing a preset generative document image extraction model, and there are some common problems with the generative document image extraction model, such as: the result of the generative document image extraction model may have a text recognition error problem. This common problem is called information extraction defect information.

[0154] In step S504 of some embodiments, the target character recognition data is obtained by processing a preset optical character recognition model, and the optical character recognition model has some common problems, such as: the result of the optical character recognition model may have a field disconnection problem. This common problem is called character recognition defect information.

[0155] In some embodiments, the information extraction defect information and the character recognition defect information are collectively referred to as multi-source result description information;

[0156] In step S505 of some embodiments, the target output slot information and the multi-source result description information (information extraction defect information and character recognition defect information) may be filled into a preset instruction template to obtain a model configuration operation instruction.

[0157] For example, when the target document type is a bill type,

[0158] The target output slot information includes: bill number, date, transaction amount, payer, payee, and transaction method;

[0159] The multi-source result description information includes: the results of the generative document image extraction model may have text recognition errors, and the results of the optical character recognition model may have field disconnection problems.

[0160] The preset indication template is: Multi-source result description information + Please consider the target information extraction data and information extraction defect information given above, and output the following fields: + Target output slot information;

[0161] That is, the model configuration operation instructions are: the results of the generative document image extraction model may have text recognition errors, and the results of the optical character recognition model may have field disconnection problems. Please comprehensively consider the target information extraction data and information extraction defect information given above, and output the following fields: bill number, date, transaction amount, payer, payee, and transaction method.

[0162] In some embodiments, the "Please comprehensively consider the target information extraction data and information extraction defect information given above, and output the following fields: + target output slot information" in the template can be called: prediction task description, and the content of the prediction task description can be adjusted according to the actual document image information extraction task, but is not limited to this.

[0163] It can be understood that the model configuration operation instructions can be used to dynamically adjust the original information fusion model, and clearly point out possible problems and slot fields that need to be output when the results are output, so as to more accurately extract document image information and improve overall processing efficiency and accuracy.

[0164] See also Figure 6 In some embodiments, step S105 includes but is not limited to steps S601 to S602:

[0165] Step S601, obtaining a document image sample from a preset sample database based on the target document type;

[0166] Step S602: Generate an information fusion reference sample based on the document image sample and the model configuration operation instruction.

[0167] Steps S601 to S602 shown in the embodiment of the present application accurately obtain document image samples from a preset sample database based on the target document type, and generate information fusion reference samples in combination with the model configuration operation instructions. The information fusion reference samples are also used to adjust the original information fusion model, so that the model can more accurately understand the characteristics and requirements of different document types, thereby following business logic and rules in the information fusion process, ensuring that the output results meet business requirements and can effectively deal with complex situations in the document, which not only enhances the flexibility and customizability of the model, but also improves processing efficiency and reliability of results, and helps to better meet diverse business needs.

[0168] In some embodiments, the preset sample database includes multiple document image samples, all of which are pre-screened and cleaned by technical personnel to meet the needs of different business scenarios.

[0169] In step S601 of some embodiments, the document image sample includes a sample document image and expected document image information, the sample document image contains the sample document information, and the expected document image information is used to represent the document image information expected to be extracted from the sample document image.

[0170] In one embodiment, it is expected that the text image information can be collated for the technician.

[0171] See also Figure 7 In some embodiments, step S602 may include but is not limited to steps S701 to S703:

[0172] Step S701, performing generative information extraction on sample document information in a sample document image to obtain sample information extraction data;

[0173] Step S702, performing optical character recognition on the sample document information in the sample document image to obtain sample character recognition data;

[0174] Step S703, combining the sample information extraction data, the sample character recognition data, the model configuration operation instruction and the expected document image information to obtain an information fusion reference sample.

[0175] Steps S701 to S703 shown in the embodiment of the present application perform generative information extraction on sample document information in a sample document image to obtain sample information extraction data, perform optical character recognition on the sample document information in the sample document image to obtain sample character recognition data, and combine the sample information extraction data, sample character recognition data, model configuration operation instructions, and expected document image information to obtain an information fusion reference sample, thereby constructing a comprehensive and specific reference standard to adjust the original information fusion model so that the model can more accurately understand the characteristics and requirements of different document types, thereby improving processing efficiency and the reliability of results, and helping to better meet diverse business needs.

[0176] In step S701 of some embodiments, sample document information in a sample document image is subjected to generative information extraction using a preset generative document image extraction model to obtain sample information extraction data;

[0177] It should be noted that the specific implementation of the above step S701 is basically the same as the specific implementation of the above step S102, and will not be repeated here.

[0178] In step S702 of some embodiments, optical character recognition is performed on sample document information in the sample document image using a preset optical character recognition model to obtain sample character recognition data;

[0179] It should be noted that the specific implementation of the above step S702 is basically the same as the specific implementation of the above step S103, and will not be repeated here.

[0180] In some embodiments, when the target document type is a property ownership certificate type,

[0181] The sample information extraction data is: {'Property Certificate Code': 'Jin (2018) xx City Real Estate Right No. 0113122'...'Location': 'Times Qiongqiao Plaza Building 1 No. 0101'...};

[0182] The sample character recognition data is: {'Jin (2018', ') xx City', 'Real Estate No.', '0113122'... 'Located', 'Times Gallery Plaza Building 1 No. 0101'...};

[0183] The model configuration operation instructions are: the results of the generative document image extraction model may have text recognition errors, and the results of the optical character recognition model may have field disconnection problems. Please comprehensively consider the target information extraction data and information extraction defect information given above, and output the following fields: property certificate number, property owner, location, area, and holding time.

[0184] The expected document image information is: {'Property Certificate Code': 'Jin (2018) No. 0113122 of Real Estate Rights in xx City'... 'Location': 'No. 0101, Building 1, Times Bridge Square'...};

[0185] Therefore, by combining the above sample information extraction data, sample character recognition data, model configuration operation instructions, and expected document image information, the content obtained is the information fusion reference sample.

[0186] In some embodiments, the preset original information fusion model can be an open-source Chinese large model, such as Baichuan, Tongyi Qianwen, ChatGLM, etc.

[0187] In step S106 of some embodiments, by inputting the model configuration operation instructions and the information fusion reference sample into the original information fusion model for model transfer learning, the configuration of the model is completed, and thus a target information fusion model with the capabilities required by the business is obtained.

[0188] In some embodiments S107, the target information fusion model performs information fusion on the target information extraction data and the target character recognition data, and combines the instructions provided by the model configuration operation instructions to guide the target information fusion model to comprehensively consider the information from these two different sources of the target information extraction data and the target character recognition data, thereby making up for the potential errors of a single model and improving the accuracy of document image information extraction.

[0189] In one embodiment, if the target document type of the original document image is the property rights certificate type, the data used includes:

[0190]

Target information extraction data: {'Property Certificate Code': 'Yue (2018) No. 0033156 of Real Estate Rights in xx City'... 'Location': 'Building 1, Area A, Longcheng Villa Half County'...}

[0191]

Target character recognition data: {'Yue (', '2017)', 'xx City Real Estate Rights No.', '0033156'... 'Location', 'Building 1, Area A, Longcheng Villa Guan County'...}

[0192]

The model configuration operation instruction is: There may be problems with character recognition errors in the results of the generative document image extraction model, and there may be problems with field disconnection in the results of the optical character recognition model. Please comprehensively consider the above-given target information extraction data and information extraction defect information, and output the following fields: Property Certificate Number, Property Owner, Location, Area, Holding Time

[0193] 【Information fusion reference sample:

[0194] {The sample information extraction data is: {'Property certificate code': 'Jin (2018) xx City Real Estate No. 0113122'...'Location': 'Times Qiongqiao Plaza Building 1 No. 0101'...};

[0195] The sample character recognition data is: {'Jin (2018', ') xx City', 'Real Estate No.', '0113122'... 'Located', 'Times Gallery Plaza Building 1 No. 0101'...};

[0196] The model configuration operation instructions are: the results of the generative document image extraction model may have text recognition errors, and the results of the optical character recognition model may have field disconnection problems. Please consider the target information extraction data and information extraction defect information given above, and output the following fields: property certificate number, property owner, location, area, and holding time;

[0197] The expected document image information is: {'Property Certificate Code': 'Jin (2018) xx City Real Estate Right No. 0113122'... 'Location': 'Times Gallery Plaza Building 1 No. 0101'...}}];

[0198] By inputting the model configuration operation instructions and the information fusion reference samples into the original information fusion model for model transfer learning, the model configuration is completed and the target information fusion model is obtained;

[0199] The target information extraction data and the target character recognition data are fused through the target information fusion model to obtain the target document image information;

[0200] Among them, the target document image information is: {'Property certificate code': 'Guangdong (2018) xx City Real Estate Right No. 0033156'...'Location': 'Building 1, Zone A, Guanjun, Longcheng Villa'...}.

[0201] See also Figure 8 The embodiment of the present application also provides a document image information extraction device, which can implement the above document image information extraction method, and the device includes:

[0202] The document image acquisition module 801 is used to acquire the original document image; wherein the original document image contains the original document information;

[0203] A generative information extraction module 802 is used to perform generative information extraction on the original document information in the original document image to obtain target information extraction data;

[0204] The optical character recognition module 803 is used to perform optical character recognition on the original document information in the original document image to obtain target character recognition data;

[0205] An operation instruction determination module 804, for determining a model configuration operation instruction based on the original document image;

[0206] A reference sample determination module 805 is used to determine an information fusion reference sample according to the original document image and the model configuration operation instruction;

[0207] The model configuration module 806 is used to configure the preset original information fusion model according to the model configuration operation instruction and the information fusion reference sample to obtain the target information fusion model;

[0208] The information fusion module 807 is used to fuse the target information extraction data and the target character recognition data through a target information fusion model to obtain target document image information.

[0209] The specific implementation of the document image information extraction device is basically the same as the specific implementation of the document image information extraction method described above, and will not be described in detail here.

[0210] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the document image information extraction method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0211] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0212] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0213] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the document image information extraction method of the embodiment of this application;

[0214] Input / output interface 903, used to implement information input and output;

[0215] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0216] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0217] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0218] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned document image information extraction method is implemented.

[0219] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0220] The document image information extraction method and device, electronic device and storage medium provided by the embodiments of the present application obtain an original document image; wherein the original document image contains original document information; generative information extraction is performed on the original document information in the original document image to obtain target information extraction data, and the complex layout and context information of the document can be learned; optical character recognition is performed on the original document information in the original document image to obtain target character recognition data, thereby ensuring accurate recognition of text content; model configuration operation instructions are determined based on the original document image; information fusion reference samples are determined according to the original document image and the model configuration operation instructions; a preset original information fusion model is configured according to the model configuration operation instructions and the information fusion reference samples to obtain a target information fusion model; information fusion is performed on the target information extraction data and the target character recognition data through the target information fusion model to obtain more accurate and complete target document image information, thereby effectively improving the comprehensiveness, accuracy and efficiency of document image information extraction.

[0221] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0222] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0223] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0224] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0225] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0226] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0227] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0228] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0229] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0230] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0231] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A document image information extraction method, characterized in that: The method comprises: Acquire an original document image; wherein the original document image contains original document information; Performing generative information extraction on the original document information in the original document image to obtain target information extraction data; Performing optical character recognition on the original document information in the original document image to obtain target character recognition data; Determining a model configuration operation instruction based on the original document image; Determine an information fusion reference sample according to the original document image and the model configuration operation instruction; Performing model configuration on a preset original information fusion model according to the model configuration operation instruction and the information fusion reference sample to obtain a target information fusion model; The target information extraction data and the target character recognition data are fused through the target information fusion model to obtain target document image information.

2. The method according to claim 1, characterized in that The step of determining the model configuration operation instruction based on the original document image includes: Performing type recognition on the original document image to obtain a target document type; Determine target output slot information according to the target document type; Performing defect detection on the target information extraction data to obtain information extraction defect information; Performing defect detection on the target character recognition data to obtain character recognition defect information; The information extraction defect information, the character recognition defect information and the target output slot information are filled into a preset instruction template to obtain the model configuration operation instruction.

3. The method according to claim 2, characterized in that The determining of the information fusion reference sample according to the original document image and the model configuration operation instruction includes: Acquiring a document image sample from a preset sample database based on the target document type; The information fusion reference sample is generated based on the document image sample and the model configuration operation instruction.

4. The method according to claim 3, characterized in that The document image sample includes a sample document image and expected document image information, wherein the sample document image contains sample document information, and the expected document image information is used to represent document image information expected to be extracted from the sample document image; and generating the information fusion reference sample based on the document image sample and the model configuration operation instruction includes: Performing generative information extraction on the sample document information in the sample document image to obtain sample information extraction data; Performing optical character recognition on the sample document information in the sample document image to obtain sample character recognition data; The sample information extraction data, the sample character recognition data, the model configuration operation instruction and the expected document image information are combined to obtain the information fusion reference sample.

5. The method according to claim 1, characterized in that: The performing generative information extraction on the original document information in the original document image to obtain target information extraction data includes: Performing feature encoding on the original document image to obtain document image features; Feature decoding is performed on the document image features to obtain the target information extraction data.

6. The method according to claim 5, characterized in that The step of decoding the document image features to obtain the target information extraction data includes: Performing attention calculation on the document image features to obtain image text distribution features; Performing context calculation on the image text distribution features to obtain original text features; Performing text prediction on the original text features to obtain text prediction probability data; Determining target text information based on the text prediction probability data; The target information extraction data is generated according to the target text information.

7. The method according to any one of claims 1 to 6, characterized in that: The obtaining of the original document image comprises: Obtaining an initial document image; Performing denoising processing on the initial document image to obtain a denoised document image; Performing image enhancement processing on the denoised document image to obtain an enhanced document image; The enhanced document image is resized to obtain the original document image.

8. A document image information extraction device, characterized in that: The device comprises: A document image acquisition module, used to acquire an original document image; wherein the original document image contains original document information; A generative information extraction module, used for performing generative information extraction on the original document information in the original document image to obtain target information extraction data; An optical character recognition module, used for performing optical character recognition on the original document information in the original document image to obtain target character recognition data; An operation instruction determination module, used for determining a model configuration operation instruction based on the original document image; A reference sample determination module, used to determine an information fusion reference sample according to the original document image and the model configuration operation instruction; A model configuration module, used to configure a preset original information fusion model according to the model configuration operation instruction and the information fusion reference sample to obtain a target information fusion model; The information fusion module is used to fuse the target information extraction data and the target character recognition data through the target information fusion model to obtain the target document image information.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images