Invoice identification method, device and equipment based on multi-modal large model

Abstract

The invention relates to the technical field of image processing, and provides an invoice recognition method, device and equipment based on a multi-modal large model, and the method comprises the steps: generating the global semantic description of a to-be-processed invoice image through the multi-modal large model, and carrying out the recognition of the to-be-processed invoice image through the global semantic description. Performing region instance segmentation on the to-be-processed invoice image by using the multi-modal large model, outputting an independent bill sub-image corresponding to each bill, constructing a multi-modal guide instruction according to the global semantic description and a preset structured template, and then inputting the independent bill sub-images and the corresponding guide instruction into the multi-modal large model, and obtaining invoice information structured data, wherein the invoice information structured data can directly meet preset requirements. And the invoice information structured data is corrected through the rigid text features extracted by the optical character recognition model, so that the global reasoning advantage of the multi-modal large model is reserved, and the accuracy of the invoice information structured data is further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an invoice recognition method, device and equipment based on a multi-modal large model. BACKGROUND

[0002] With the development of information technology, electronic invoices are widely used. The commonly used management method of input electronic invoices is to upload the invoices to a computer system by taking pictures, and then read the text information in the target area of the invoice to be recognized by dividing it into multiple areas or through an OCR (Optical Character Recognition) text recognition method, to obtain the target text, and then integrate the information on the ticket surface obtained from the target text into a table for output. In this way, the OCR technology can only extract partial text of the invoice, lacks semantic structuring ability, requires manual secondary arrangement of field mapping relationship, and is easily affected by the subjectivity of financial personnel, with poor accuracy. SUMMARY

[0003] The present application provides an invoice recognition method, device and equipment based on a multi-modal large model, so that the structured data of the invoice information obtained by invoice recognition can directly meet the preset requirements, and the accuracy of the structured data of the invoice information is improved.

[0004] The invoice recognition method based on a multi-modal large model provided by the present application is characterized by comprising:

[0005] obtaining an invoice image to be processed;

[0006] In the case where the invoice image to be processed is a multi-invoice splicing image, a global semantic description of the invoice image to be processed is generated by a multi-modal large model, and region instance segmentation is performed on the invoice image to be processed by the multi-modal large model to output an independent invoice sub-image corresponding to each invoice;

[0007] A multi-modal guidance instruction corresponding to the independent invoice sub-image is constructed, and the multi-modal guidance instruction is generated according to the global semantic description and a preset structured template;

[0008] The independent invoice sub-image and the corresponding guidance instruction are input into the multi-modal large model to generate invoice information structured data, and the invoice information structured data is used to describe the semantic content of the independent invoice sub-image;

[0009] The rigid text features of the independent invoice sub-image are extracted based on an optical character recognition model, and the invoice information structured data is corrected based on the rigid text features.

[0010] In this way, the invoice recognition method based on the multi-modal large model provided by the embodiment of the present application generates the global semantic description of the to-be-processed invoice image through the multi-modal large model, performs regional instance segmentation on the to-be-processed invoice image by using the multi-modal large model, outputs an independent invoice sub-image corresponding to each invoice, constructs a multi-modal guide instruction according to the global semantic description and the preset structured template, then inputs the independent invoice sub-image and the corresponding guide instruction into the multi-modal large model, and obtains invoice information structured data, which can directly meet the preset requirements. Moreover, the rigid text features extracted by the optical character recognition model are used to correct the invoice information structured data, thereby further improving the accuracy of the invoice information structured data while retaining the global reasoning advantage of the multi-modal large model.

[0011] In a possible implementation, the rigid text features include digital information, a position region corresponding to the digital information, and a digital confidence degree.

[0012] The step of correcting the invoice information structured data based on the rigid text features includes:

[0013] According to the position region corresponding to the digital information, the corresponding comparison information in the invoice information structured data is determined.

[0014] In a case where the digital information is inconsistent with the comparison information and the digital confidence degree is greater than or equal to a first confidence threshold, the comparison information of the invoice information structured data is covered by the digital information.

[0015] In a case where the digital information is inconsistent with the comparison information and the digital confidence degree is greater than or less than a second confidence threshold and less than the first confidence threshold, the digital information is marked in a corresponding field in the invoice information structured data, and the second confidence threshold is less than the first confidence threshold.

[0016] In this way, the recognition accuracy of the OCR model for rigid numbers is high, which can make up for the insufficient recognition accuracy of the multi-modal large model for rigid numbers, thereby effectively solving the fine-grained defects of number recognition while retaining the global reasoning advantage of the multi-modal large model and introducing the local rigid constraint of the OCR model.

[0017] In a possible implementation, the rigid text features include character-level glyph features, and the step of correcting the invoice information structured data based on the rigid text features includes:

[0018] The character-level glyph features are arbitrated based on the large language model to be consistent with the semantics of the invoice information structured data.

[0019] In a case where the character-level glyph feature and the invoice information structured data exist semantic inconsistency, and the confidence of the character-level glyph feature output by the large language model is greater than the third confidence threshold, the character-level glyph feature is used to cover the text in the invoice information structured data corresponding to the character-level glyph feature.

[0020] In this way, while retaining the global inference advantage of the multi-modal large model, the glyph-semantic-based verification mechanism solves the problem of semantic distortion of the multi-modal large model in multi-lingual invoice recognition.

[0021] In a possible implementation, the rigid text features include spatial correlation relationships between text blocks.

[0022] The step of correcting the invoice information structured data based on the rigid text features includes:

[0023] In a case where the spatial correlation relationship between the two text blocks is inconsistent with the invoice information structured data, and the spatial correlation relationship between the two text blocks is determined to be a strong spatial correlation, the corresponding field mapping content in the invoice information structured data is adjusted according to the correlation relationship between the two text blocks.

[0024] In this way, the spatial correlation relationship between text blocks is obtained by using the OCR model, and when the text entries on the invoice are across lines or exist up and down floating, the mapping relationship between the goods and the prices can be accurately reconstructed, thereby improving the accuracy of the invoice information structured data.

[0025] In a possible implementation, the spatial correlation relationship between the two text blocks is determined to be a strong spatial correlation, specifically: the Euclidean distance between the two text blocks satisfies a preset spatial relationship condition.

[0026] In a possible implementation, the preset spatial relationship condition is that the horizontal offset is less than a first distance threshold and the center line vertical offset is less than a second distance threshold.

[0027] In a possible implementation, after the step of obtaining the invoice image to be processed, the method further includes:

[0028] Extracting text features of the invoice image to be processed;

[0029] Identifying the language of the text features based on a language detection algorithm;

[0030] According to the language, a guided instruction template of the multi-modal large model is called.

[0031] In this way, the instruction template calling mechanism is adapted to the language, thereby improving the structured recognition accuracy of invoices across languages.

[0032] In a possible implementation, after the step of performing region instance segmentation on the invoice image to be processed by using the multi-modal large model to output an independent invoice sub-image corresponding to each invoice, the method further includes: preprocessing the independent invoice sub-image, and the preprocessing includes rotation correction, boundary cropping, and / or color equalization.

[0033] In this way, the efficiency of subsequent recognition of the independent invoice sub-image can be improved

[0034] The invoice recognition device based on the multi-modal large model provided in the embodiments of the present application includes:

[0035] The obtaining module is configured to obtain an invoice image to be processed.

[0036] The multi-modal large model calling module is configured to, in a case where the invoice image to be processed is a multi-invoice splicing image, generate a global semantic description of the invoice image to be processed by using the multi-modal large model, and perform region instance segmentation on the invoice image to be processed by using the multi-modal large model to output an independent invoice sub-image corresponding to each invoice.

[0037] The guidance instruction construction model is configured to construct a multi-modal guidance instruction corresponding to the independent invoice sub-image, and the multi-modal guidance instruction is generated according to the global semantic description and a preset structured template.

[0038] The multi-modal large model calling module is further configured to input the independent invoice sub-image and the corresponding guidance instruction into the multi-modal large model to generate invoice information structured data, and the invoice information structured data is used to describe the semantic content of the independent invoice sub-image.

[0039] The correction module is configured to extract rigid text features of the independent invoice sub-image based on an optical character recognition model, and correct the invoice information structured data based on the rigid text features.

[0040] In this way, the invoice recognition device based on the multi-modal large model provided in the embodiments of the present application generates a global semantic description of the invoice image to be processed by using the multi-modal large model, performs region instance segmentation on the invoice image to be processed by using the multi-modal large model to output an independent invoice sub-image corresponding to each invoice, constructs a multi-modal guidance instruction according to the global semantic description and a preset structured template, then inputs the independent invoice sub-image and the corresponding guidance instruction into the multi-modal large model, and corrects the invoice information structured data by using rigid text features extracted by an optical character recognition model, thereby further improving the accuracy of the invoice information structured data while retaining the global reasoning advantage of the multi-modal large model.

[0041] The embodiment of the application provides an invoice recognition device based on a multimodal large model, which comprises a processor, a memory, a communication interface and a communication bus, and the processor, the memory and the communication interface complete mutual communication through the communication bus.

[0042] The memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation of the invoice recognition method based on the multimodal large model.

[0043] In this way, the invoice recognition device based on the multimodal large model provided by the embodiment of the application, when the processor executes the program, generates the global semantic description of the to-be-processed invoice image through the multimodal large model, performs regional instance segmentation on the to-be-processed invoice image by using the multimodal large model, outputs an independent invoice sub-image corresponding to each invoice, constructs a multimodal guide instruction according to the global semantic description and a preset structured template, then inputs the independent invoice sub-image and the corresponding guide instruction into the multimodal large model, and corrects invoice information structured data through rigid text features extracted by an optical character recognition model, thereby further improving the accuracy of the invoice information structured data while retaining the global reasoning advantage of the multimodal large model. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0045] Figure 1 Flowchart of the invoice recognition method based on the multimodal large model provided by some embodiments of the application Figure 1 ;

[0046] Figure 2 Schematic diagram of a multi-invoice splicing image provided by some embodiments of the application Figure 1 ;

[0047] Figure 3 Schematic diagram of a multi-invoice splicing image provided by some embodiments of the application Figure 2 ;

[0048] Figure 4 Schematic diagram of a single invoice image after regional instance segmentation provided by some embodiments of the application

[0049] Figure 5 Flowchart of the invoice recognition method based on the multimodal large model provided by some embodiments of the application Figure 2 ;

[0050] Figure 6 Flowchart of the invoice recognition method based on the multi-modal large model provided by some embodiments of the present application Figure 3 ;

[0051] Figure 7 Flowchart of the invoice recognition method based on the multi-modal large model provided by some embodiments of the present application Figure 4 ;

[0052] Figure 8 Flowchart of the invoice recognition method based on the multi-modal large model provided by some embodiments of the present application Figure 5 ;

[0053] Figure 9 Structure diagram of the invoice recognition model based on the multi-modal large model provided by some embodiments of the present application. DETAILED DESCRIPTION

[0054] The embodiments will be described in detail below with reference to examples thereof as illustrated in the accompanying drawings. In the following description, same numbers refer to same or similar elements throughout the drawings. The embodiments described in the following examples do not represent all the implementations consistent with the present application. They are merely examples of systems and methods consistent with some aspects of the present application as recited in the detailed description of the claims.

[0055] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0056] The terms "first", "second", "third", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar or like objects or entities, and do not necessarily mean a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0057] The terms "include" and "have" and any variations thereof are intended to cover but not exclusive inclusion, for example, a product or device including a series of components does not necessarily limit to all components clearly listed, but can include other components not clearly listed or inherent to these products or devices.

[0058] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0059] The embodiments of the present application provide an invoice recognition method based on a multi-modal large model, asFigure 1 The invoice recognition method comprises S110 to S150.

[0060] S110, an image of a to-be-processed invoice is acquired.

[0061] The image of the to-be-processed invoice can be an electronic document or an electronic image. For example, an electronic invoice stored in an image format. For another example, an electronic invoice stored in a document form. The image of a paper invoice acquired by a camera or a scanning device can also be obtained.

[0062] S120, in a case where the image of the to-be-processed invoice is a multi-invoice splicing image, a global semantic description of the image of the to-be-processed invoice is generated by using a multimodal large model, and regional instance segmentation is performed on the image of the to-be-processed invoice by using the multimodal large model, and an independent invoice sub-image corresponding to each invoice is output.

[0063] The image of the to-be-processed invoice processed by the technical solution of the present application is a multi-invoice splicing image. For example, in an actual application process, multiple paper invoices are pasted in a predetermined area. Thus, when the image of the multiple paper invoices in the predetermined area is acquired, the acquired invoice image is a multi-invoice splicing image. For another example, multiple electronic invoices are combined into one image.

[0064] In order to enable a multimodal large model (MLLM) to more accurately generate a global semantic description of the image of the to-be-processed invoice, the global semantic description refers to a comprehensive text representation generated by the multimodal large model after high-level semantic understanding of the overall content of the multi-invoice splicing image. For example, the global semantic description includes overall layout description, invoice type identification, key field summary, and context semantic association. The overall layout description is to identify the spatial arrangement relationship of the multiple invoices (such as "two receipts on the left and three tickets on the right"). The invoice type identification is to classify the semantic categories of various invoices (such as "receipt / ticket / value-added tax invoice"). The key field summary is to extract the core business information across invoices (such as "total amount 2580 Japanese yen, date April 27, 2024"). The context semantic association is to reveal the business relationship between the invoices (such as "two receipts are issued by the same merchant").

[0065] In an embodiment of the present application, an instruction dataset is constructed to fine-tune the multimodal large model to optimize the semantic understanding ability of the multimodal large model. For example, the GPT-4O model is used to construct an instruction dataset, and a specific guidance instruction template is set according to the instruction dataset, and then the GPT-4O model is used to perform detailed content recognition on the mixed invoices in the multi-bill splicing image. The guidance instruction template contains the key fields that need to be extracted from the invoice, such as invoice number, invoice date, amount, tax number, etc. These suggestive guidance instructions guide the multimodal large model to focus on specific areas of the invoice image to be processed, ensuring that the required information can be accurately extracted.

[0066] For example, Figure 2 The invoice content is constructed in the following format:

[0067] {"messages":[{"role":"user","content":" A detailed description of the invoice image to be processed from a global perspective."},{"role":"assistant","content":"This invoice image to be processed shows two white paper receipts and three orange tickets. The two receipts on the left have similar contents, both of which are transaction records for the purchase of "Lianer Tickets" for an amount of 2,580 yen, dated April 27, 2024, and April 7, 2024, respectively. The receipts detail the tax and total price of the purchase, and are accompanied by a transaction number and instructions for use. The three orange tickets on the right have black text printed on them, indicating the date and time of use, the ticket number, the fare, and related transportation information. These tickets and receipts are neatly arranged together against a light gray background, with a clear overall image and readable text. "}],"images":[" / xxx / x.jpg"]}

[0068] Similarly, in order for the multimodal large model to more accurately output the independent bill sub-image corresponding to each bill, a regional dataset of multiple bills is constructed, and the semantic understanding ability of the multimodal model is fine-tuned and optimized based on the instruction data. Specifically, in the process of constructing the regional dataset of multiple bills, data labelers conduct a detailed analysis of each multi-bill spliced ​​image to identify the boundaries and position of each invoice. For example, through image labeling tools, the labelers manually draw the detection area of ​​each invoice on the image to ensure that the boundaries of each area are clear and accurate. Then, corresponding labels are assigned to each detection area. These labels usually include key field information of the invoice. After completing the manual labeling, the instruction data for fine-tuning the multimodal large model is constructed.

[0069] For example, Figure 3 The invoice content and instruction data format are as follows:

[0070] {"messages":[{"role":"system","content":"You are a helpfulassistant."},{"role":"user","cont ent":" box out the <|object_ref_start|>invoice<|object_ref_end|> in the image"},{"role":"assistant","cont ent":"<|box_start|>(7,16),(299,647)<|box_end|><|box_start|>(310,20),(570,645)<|box_end|><|box_start|>(579,27),(896,235)<|box_end|><|box_start|>(583,245),(898,440)<|box_end|><|box_start|>(582,449),(896,648)<|box_end|>"}],"images":[" / xxx / x.jpg"]}。

[0071] S130, construct a multi-modal guidance instruction corresponding to the independent invoice sub-image, and the multi-modal guidance instruction is generated according to the global semantic description and a preset structured template.

[0072] Wherein, after obtaining the global semantic description, the basic content of the plurality of invoices in the to-be-processed invoice image can be preliminarily determined. In actual application process, the user pays attention to different key contents for different types of invoices. For example, for a catering invoice, the user pays attention to dish information, price and whether to contain non-dish service charge. For example, for an invoice used for enterprise reimbursement, the user pays attention to identifying the purchase information.

[0073] On this basis, in order to make the information that the user pays attention to be described in detail, the present application determines a preset structured template according to the global semantic description, and constructs a multi-modal guidance instruction corresponding to the independent invoice sub-image based on the global semantic description and the preset structured template. So as to make the multi-modal large model output description content more in line with the requirements of the user by using the multi-modal guidance instruction corresponding to the independent invoice sub-image.

[0074] Wherein, after the step of outputting the independent invoice sub-image corresponding to each invoice, the independent invoice sub-image needs to be preprocessed, so as to further process the independent invoice sub-image by the multi-modal large model and the OCR model, wherein the preprocessing includes rotation correction, boundary cropping and / or color equalization. In this way, the efficiency of subsequent recognition of the independent invoice sub-image can be improved.

[0075] S140: Input the independent bill sub-image and the corresponding guidance instruction into the multimodal large model to generate invoice information structured data, where the invoice information structured data is used to describe the semantic content of the independent bill sub-image.

[0076] In order to enable the multimodal large model to more accurately describe the content of the independent bill sub-image. In an embodiment of the present application, it is necessary to fine-tune and optimize the semantic understanding ability of the multimodal model based on the instruction data. Specifically, first, the GPT-4O model will identify the independent invoice area based on pre-set prompt words. These prompt words usually include key fields of the invoice, such as invoice number, invoice date, amount, tax number, etc. During the recognition process, GPT-4O uses its powerful natural language processing and image recognition capabilities to parse the text information in the invoice. The model can understand the content of invoices in different formats and languages ​​and convert them into structured data. These data are usually output in the form of formatted JSON to ensure that the information is clear and easy to process, and then construct the model to fine-tune the instruction data.

[0077] For example, Figure 4 The invoice content, construction instruction data format is as follows:

[0078] {"messages":[{"role":"user","content":"Based on the document image provided by the user and the OCR content of the image, identify the document type, extract and record the following information:\n-docType: Document type, identify whether the document is an invoice file. Invoice files record transactions and can be used as original documents for accounting, payment, and tax processing; they include invoices, receipts, slips, bills, deduction notices, receipts, bills, invoice lists, etc. If it is any of the above types, return 'invoice'; otherwise, return 'other'.\n-nameOfInvoice: Invoice title, the title on the invoice, such as 'VATInvoice', 'Debit Note', etc.\n-invoiceNumber: Invoice number or receipt number. If it cannot be found, search for 'Invoice' on the invoice. No', 'Request Book Number', 'Request Number', 'Sheet No', 'Voucher No', 'Declaration Number', 'No.', 'No'. \n-invoiceCode: Invoice type code. If there is no invoice type code information, then search for the invoice serial number, 'Serial', Not invoice type code. \n-invoiceDate: Invoice issue date. \n-totalNetAmount: Total amount without tax. \n-totalAmount: Total amount. \n-totalTaxAmount: Total tax amount, the sum of all tax amounts. \n-currency: Currency, output in ISO 4217 currency code form, such as: USD, CNY, CAD, AUD, GBP, JPY, DEM, HKD, FRF, CHF, VND, etc. \n-billToName: The name of the recipient. If there is no 'Bill To', find 'MESSRS', 'Purchaser', 'Customer', 'Buyer', 'Attention to' or other words synonymous with 'purchaser'. \n-billToComposite: The complete address information of the recipient. \n-billToCountry: The country where the recipient is located, extracted from the complete address of the recipient. \n-billToTaxIdentificationNumber: The tax registration code of the recipient. \n-shipFromComposite: The shipping address. \n-billFromName: The name of the invoice issuer. If there is no 'From' information, or the 'From' information is not like a company name, find: 'Account Name', 'Beneficiary Name', 'Seller', 'Remit to', the signature at the bottom, or the title, receipt or small ticket of the merchant name. \n-billFromComposite: The complete address information of the invoice issuer. \n-billFromCountry: The country where the invoice issuer is located, extracted from the address information of the invoice issuer. \n-billFromTaxIdentificationNumber: The tax identification number of the invoice issuer. If not found, find the tax registration code, 'VAT Registration', 'TAX#'. \n-purchaseOrderNumber: PO number on the invoice. \n-dueDate: Payment deadline.-detailOfGoodsOrServices: Details of the goods or services in the invoice, including the name, remarks, amount excluding tax, tax rate, tax amount, amount including tax, and order number of each good or service, extracted in the following format: json[{\n \"articleName\":\"Goods or Services Name (complete information needs to be extracted)\",\n \"description\":\"Remarks\",\n \"netAmount\":\"Amount excluding tax\",\n \"taxRate\":\"Tax rate\",\n \"tax\":\"Tax amount\",\n \"grossAmount\":\"Amount including tax\",\n \"orderNumber\":\"Order number (note number)\"}\n] -detailOfTaxSummar y: Summary of taxes in the invoice, including tax type, tax rate, tax base (amount to be calculated / amount excluding tax), and tax amount. Please note that the same tax type but different tax rates need to be extracted separately. Extract in the following format: \n```json\n[\n{\n.

[0079] \"taxCategory\":\"Tax type (such as 'vat', 'Consumption Tax')\",\n\"taxRate\":\"Tax rate\",\n\"netTaxableAmount\":\"Tax base\",\n\"tax\":\"Tax amount\"\n}\n]\n```\n\nNote:\n- Extract information using a combination of the document image and OCR results. OCR results may contain errors, requiring comprehensive judgment based on the document image and automatic correction.\n- Do not automatically translate the extracted content; retain the original text.\n- When extracting amount-related content, including but not limited to the total amount excluding tax, total amount, total tax amount, amount excluding tax, tax amount, amount including tax, and tax base, extract it as a number without currency separators.\n- When extracting time-related content, convert it to the 'YYYY-MM-DD' format.\n- When extracting tax rates, output them as a percentage, for example, '10%'.\n- The 'detail' on receipts and receipts In the 'OfGoodsOrServices' information, it's necessary to determine whether the detailed amount is tax-inclusive by analyzing whether the total amount includes tax. The 'Registration Number' is the tax identification number, not the invoice number. The 'Registration Number' on receipts and slips is generally 'billFromTaxIdentificationNumber'. The 'currency' field must be output. If the invoice does not have currency information, one is inferred. For other fields, especially those related to amounts (such as total amount excluding tax, total amount, total tax amount, etc.), if the corresponding information cannot be found, do not calculate or supplement it yourself; instead, return None. Only the JSON result is output, without other text. The JSON key only contains 'docType', 'nameOfInvoice', 'invoiceNumber', 'invoiceCode', 'invoiceDa te','totalNetAmount','totalAmount','totalTaxAmount','currency','billToName','billToComposite','billT oCountry','billToTaxIdentificationNumber','shipFromComposite','billFromName','billFromComposi te','billFromCountry','billFromTaxIdentificationNumber','purchaseOrderNumber','dueDate','detailOfGoodsOrServices','detailOfTaxSummary'."}]}]}},{"role":"assistant","content":{"docType":"invoic e","nameOfInvoice":"Receipt","invoiceNumber":null,"invoiceCode":null,"invoiceDate":"2024-04-27","totalNetAmount":null,"totalAmount":2580,"totalTaxAmount":null,"currency":"JPY ","billToName":null,"billToComposite":null,"billToCountry":null,"billToTaxIdentificationNumber":null,"shipFromComposite":null,"billFromName":"Billing Party Name","billFromComposite":"Shiratai Station","bi llFromCountry":null,"billFromTaxIdentificationNumber":"T7010601012155","purchaseOrderNum ber":null,"dueDate":null,"detailOfGoodsOrServices":[{"articleName":"ライナー coupon purchase","descripti on":"","netAmount":null,"taxRate":"10%","tax":null,"grossAmount":2580,"orderNumber":null}],"det ailOfTaxSummary":[{"taxCategory":"Consumption Tax","taxRate":"10%","netTaxableAmount":null,"tax":null}]}}]}.

[0080] By fine-tuning the multimodal large model for independent bill sub-image recognition, and based on the fine-tuning, using multimodal guidance instructions constructed based on global semantic descriptions and preset structured templates as prompt words for the multimodal large model, the multimodal large model can output structured invoice information data that better meets user needs.

[0081] S150, extracting rigid text features of the independent bill sub-image based on the optical character recognition model, and correcting the invoice information structured data based on the rigid text features.

[0082] When a large multimodal model recognizes the invoice content of an independent bill sub-image, it may have recognition errors in low resolution, similar fonts, or cross-language scenarios due to the special scenarios of the invoice content.

[0083] For example, when the multimodal large model is identifying invoice content, there is a bottleneck in number recognition. The root cause is that the visual attention mechanism is unable to distinguish characters with similar glyphs. For example, for the numbers 3 and 8, and for the numbers 7 and 1, because the local features of the number glyphs are highly similar, they are prone to misjudgment in low-resolution or occluded scenarios. In addition, because the feature extraction layer of the multimodal large model has limited sensitivity to tiny text, it often leads to missed recognition of continuous numeric strings. For example, the amount: ￥5,800 misses the thousandths comma and becomes "Amount: ￥5800". Such defects result in a high recognition error rate for key fields of the invoice content (amount, date, invoice code), which seriously restricts the reliability of the structured data of the invoice information.

[0084] For example, large multimodal models have the problem of semantic distortion in multilingual invoice recognition, which is typically manifested by incorrectly simplifying Japanese (such as 営業所) into Chinese (such as 业务所).

[0085] For example, when a multimodal large model recognizes invoice content, it faces the problem of structural misalignment. This is mainly because the visual-linguistic alignment mechanism is not sensitive enough to data arrangement. When product items cross rows or fluctuate up and down, the multimodal large model will incorrectly associate the nearest price items.

[0086] In response to the above problems, in the technical solution of the present application, the rigid text features of independent bill sub-images are extracted based on the optical character recognition model, wherein the core goal of the optical character recognition model is to convert the text information in the image into editable text data, and its implementation process includes image preprocessing, word detection and positioning, character segmentation and recognition, and post-processing optimization.

[0087] Among them, image preprocessing includes denoising (eliminating interference points in the image), binarization (such as converting color or grayscale images into black and white binary images to highlight the text area), and binarization tilt correction (such as adjusting the image angle to ensure that the text is arranged horizontally).

[0088] Among them, text detection and positioning is to locate the text area in the image through convolutional neural network (CNN) or target detection algorithm (such as YOLO, EAST) and output the bounding box.

[0089] Among them, character segmentation and recognition is to split the text line into individual characters, and then use a classification model (such as CNN, LSTM) to classify each character and map it to the corresponding code.

[0090] Among them, the post-processing optimization is to correct the recognition errors (such as mistaking the number 0 as the letter O) by combining NLP technology (such as N-gram, Transformer).

[0091] In this way, the invoice recognition method based on the multi-modal large model provided in the embodiments of the present application generates a global semantic description of the to-be-processed invoice image through the multi-modal large model, performs regional instance segmentation on the to-be-processed invoice image by using the multi-modal large model, outputs an independent invoice sub-image corresponding to each invoice, constructs a multi-modal guidance instruction according to the global semantic description and a preset structured template, inputs the independent invoice sub-image and the corresponding guidance instruction into the multi-modal large model, and obtains invoice information structured data. The invoice information structured data can directly meet the preset requirements. Moreover, the rigid text features extracted by the optical character recognition model are used to correct the invoice information structured data, thereby further improving the accuracy of the invoice information structured data while retaining the global reasoning advantage of the multi-modal large model.

[0092] In a possible implementation, the rigid text features include digital information, a position region corresponding to the digital information, and a digital confidence. The step of correcting the invoice information structured data based on the rigid text features includes S510 to S530.

[0093] S510, determining corresponding comparison information in the invoice information structured data according to the position region corresponding to the digital information.

[0094] The present application provides an OCR model based on target detection and text recognition, which cooperates with the RNN model recognition through the YOLOv8 model positioning, can realize pixel-level text capture in the preset coordinate region (such as the amount frame and the date column), thereby extracting the digital information and the position region corresponding to the digital information, and outputting the digital confidence of the digital information by the RNN model. According to the position region corresponding to the digital information, the comparison information corresponding to the digital information can be determined in the invoice information structured data.

[0095] S520, in the case that the digital information is inconsistent with the comparison information and the digital confidence is greater than or equal to a first confidence threshold, using the digital information to cover the comparison information of the invoice information structured data.

[0096] In the embodiments of the present application, the first confidence threshold is 95%, that is, in the case that the digital information is inconsistent with the comparison information and the digital confidence is greater than or equal to 95%, the comparison information in the invoice information structured data is replaced by the digital information recognized by the OCR model.

[0097] S530, when the digital information is inconsistent with the comparison information and the digital confidence is greater than or equal to the second confidence threshold and less than the first confidence threshold, the digital information is marked in the corresponding field in the invoice information structured data, and the second confidence threshold is less than the first confidence threshold.

[0098] In an embodiment of the present application, the second confidence threshold is 90%, that is, when the digital information is inconsistent with the comparison information and the digital confidence is greater than or equal to 90% and less than 95%, the comparison information in the invoice information structured data is maintained unchanged, but the digital information is marked in the corresponding field in the invoice information structured data to facilitate the subsequent use of the invoice information structured data to further determine the accuracy of the content.

[0099] In this way, the OCR model has a high recognition accuracy for rigid digits, which can effectively compensate for the insufficient recognition accuracy of rigid digits by the multimodal large model. While retaining the global reasoning advantages of the multimodal large model, the introduction of the local rigid constraints of the OCR model can effectively solve the fine-grained defects of digit recognition.

[0100] In a possible implementation, the rigid text features include character-level glyph features; and the step of correcting the invoice information structured data based on the rigid text features further includes S610 to S620.

[0101] S610, arbitrating the semantic consistency between the character-level glyph features and the invoice information structured data based on the large language model.

[0102] The character-level glyph features in the independent bill sub-image are extracted through the OCR model. The character-level glyph features output by the OCR model are then combined with the invoice information structured data output by the multimodal large model and input into the Large Language Model (LLM). Using structured prompt words, semantic consistency judgment is performed by multilingual understanding capabilities.

[0103] Character-level glyph features focus on the physical structure of individual characters, rather than semantic or phonetic information. They describe the following elements through mathematical or coded methods: stroke composition (such as the three horizontal strokes in the Chinese character for wood), spatial layout (such as the upper and lower structures in Chinese characters), contour and center of gravity (such as the distance between the convex point of a character and its center of gravity), and deformation characteristics (such as tilt angle and connected stroke form).

[0104] For example, the structured prompt words are:

[0105] Please strictly compare the following text to see if it retains the original language semantics: the multimodal large model output result text A (business office) and the OCR model output result B (営業所), and determine whether the confidence level of the OCR model output result B meets the requirements.

[0106] S620, in the case that the semantic inconsistency exists between the character-level glyph feature and the invoice information structured data, and the confidence of the character-level glyph feature output by the large language model is greater than the third confidence threshold, the character-level glyph feature is used to cover the text corresponding to the character-level glyph feature in the invoice information structured data.

[0107] According to the arbitration result of the large language model, a decision is made. For example, when the LLM model determines that the semantics are inconsistent and the confidence of the OCR model output result meets the standard, the character-level glyph feature covers the text corresponding to the character-level glyph feature in the invoice information structured data. If the LLM determines that the semantics are consistent, or when the LLM determines that the semantics are inconsistent but the confidence of the OCR model output result does not meet the standard, the text corresponding to the character-level glyph feature in the invoice information structured data is maintained.

[0108] In this way, while retaining the global reasoning advantage of the multi-modal large model, the glyph-semantic-based verification mechanism solves the problem of semantic distortion of the multi-modal large model in multi-lingual invoice recognition.

[0109] In a possible implementation, the rigid text features include spatial correlation relationships between text blocks. In the case that the spatial correlation relationship between two text blocks is inconsistent with the invoice information structured data and the spatial correlation relationship between the two text blocks is determined to be strong spatial correlation, the corresponding field mapping content in the invoice information structured data is adjusted according to the correlation relationship between the two text blocks.

[0110] In the case that the multi-modal large model faces the structured mispositioning problem in recognizing invoice content, the present application proposes a correction mechanism based on the spatial relationship of adjacent texts output by the OCR model. First, the boundary box coordinates of all text blocks are extracted by the OCR model, and a two-dimensional spatial relationship matrix is constructed. Then, the Euclidean distance between the text blocks is calculated. In the case that the Euclidean distance between the two text blocks meets the preset spatial relationship condition, the spatial correlation relationship between the two text blocks is determined to be strong spatial correlation. Specifically, when the horizontal offset of the Euclidean distance is less than a first distance threshold and the center line vertical offset is less than a second distance threshold, it is determined to be strong spatial correlation. Finally, it is determined whether the spatial correlation relationship between the two text blocks in the OCR model is consistent with the invoice information structured data. In the case that the spatial correlation relationship between the two text blocks is inconsistent with the invoice information structured data and the spatial correlation relationship between the two text blocks is determined to be strong spatial correlation, the corresponding field mapping content in the invoice information structured data is adjusted according to the correlation relationship between the two text blocks. For example, the two text blocks are a commodity text block and a price text block corresponding to the commodity text block.

[0111] In this way, the spatial relationship between the text blocks is obtained by using the OCR model, and when the text entries on the invoice are across rows or have up and down floating, the mapping relationship between the commodities and the prices can be accurately reconstructed, and the accuracy of the invoice information structured data is improved.

[0112] The first distance threshold and the second distance threshold need to be determined according to the pixel size of the independent bill sub-image, for example, in the embodiment of the application, the first distance threshold is 500 pixels, and the second distance threshold is 5 pixels.

[0113] In one possible implementation, after the step of obtaining the invoice image to be processed, S710 to S730 are further included.

[0114] S710, extracting text features of the invoice image to be processed.

[0115] First, the gray processing is performed on the invoice image to be processed, the RGB color space is converted into the YUV space, and the luminance component is extracted to eliminate color interference. Then, the adaptive binarization algorithm (such as Sauvola algorithm) is used to enhance the contrast of the text region. Among them, the local threshold is dynamically calculated according to the mean and standard deviation of the pixel neighborhood. Finally, the convolutional neural network model (such as EAST text detector) is used to locate the text region in the image, and the bounding box coordinates of each text block are output. For each text block region, the following text features are extracted. Among them, the text features include content characters, position attributes and confidence values.

[0116] S720, identifying the language of the text features based on a language detection algorithm.

[0117] First, the n-gram statistical features (n takes values from 1 to 5) are extracted from the text features, the non-alphabet characters are filtered (the hyphen and apostrophe are retained), and the bag-of-words model vector is generated. Then, for the multilingual mixed text, the character encoding distribution frequency of each language is calculated to generate a language probability feature vector. Finally, the language probability feature vector is input into a bidirectional long short-term memory network classifier to output the language classification result.

[0118] The structure of the bidirectional long short-term memory network classifier includes an input layer, a hidden layer and an output layer. The dimension of the input layer is consistent with the length of the bag-of-words model feature vector. The hidden layer is a bidirectional LSTM unit that captures context dependence. The output layer includes a Softmax activation function to generate a language probability distribution.

[0119] Based on the language classification result, the language of the text features is determined. The specific language determination rule is: taking the language label with the highest probability value as the output. On this basis, if the highest probability is lower than the threshold value 0.7, it is marked as an unknown language and needs to be manually reviewed.

[0120] S730, according to the language, calling the guided instruction template of the multi-modal large model.

[0121] Different guide instruction templates store guide instruction templates corresponding to different languages. For example, a Chinese guide instruction template includes structured fields such as "extract invoice code, amount, and buyer name"; and an English guide instruction template includes fields such as "Extract Invoice Code, Amount, Buyer Name".

[0122] The guide instruction template calling logic is as follows: according to the language label output in S720, the guide instruction template is matched. If the language is an unknown language, the default template (only containing numerical fields such as invoice code and amount) is called.

[0123] For special languages (such as Arabic and Hebrew), a right-to-left layout marker can be added to the guide instruction template to ensure that the multi-modal large model parses the text in the correct direction. In addition, the order of field descriptions in the guide instruction template can be adjusted according to the grammatical characteristics of the language. For example, the guide instruction template for Chinese is structured according to the subject-predicate-object structure (such as "Buyer Name: XXX"). For example, the guide instruction template for German places the verb at the end of the instruction (such as "Rechnungsnummer: XXX ausgeben").

[0124] In this way, the cross-language invoice structure recognition accuracy is improved through the language-adaptive instruction template calling mechanism.

[0125] In order to more clearly illustrate the invoice recognition method provided by the embodiments of the present application, the whole process of the invoice recognition method of the present application is described in detail below. As shown in Figure 8 The invoice data format of the to-be-processed invoice image of the present application can be an invoice document and an invoice image.

[0126] After obtaining the to-be-processed invoice image, the language of the to-be-processed invoice image and whether the to-be-processed invoice image is a multi-invoice splicing image are recognized. The language is used as a basis for constructing a guide instruction.

[0127] In the case of a multi-invoice splicing image, the multi-modal large model is used to process the to-be-processed invoice image to obtain a global semantic description of the to-be-processed invoice image, and region instance segmentation is performed on the to-be-processed invoice image to obtain an independent invoice sub-image.

[0128] In the case of a single invoice image, the to-be-processed invoice image does not need to be cut.

[0129] The single invoice image or independent bill sub-image is subjected to text recognition by using an OCR model to obtain rigid text features, and the single invoice image or independent bill sub-image is subjected to recognition by using a multi-modal large model to obtain invoice information structured data.

[0130] Finally, the invoice information structured data is subjected to cross-validation by using the rigid text features to improve the accuracy of the invoice information structured data, and finally the invoice information structured data is output in the form of Json data.

[0131] Figure 9 The structure of an embodiment of the invoice recognition device of the present application based on a multi-modal large model is shown. As shown in the figure, Figure 8 The invoice recognition device 900 includes an acquisition module 910, a multi-modal large model calling module 920, a guide instruction construction model 930, and a correction module 940.

[0132] The acquisition module 910 is configured to acquire a to-be-processed invoice image.

[0133] The multi-modal large model calling module 920 is configured to, in a case where the to-be-processed invoice image is a multi-bill splicing image, generate a global semantic description of the to-be-processed invoice image by using a multi-modal large model, and perform regional instance segmentation on the to-be-processed invoice image by using the multi-modal large model to output an independent bill sub-image corresponding to each bill.

[0134] The guide instruction construction model 930 is configured to construct a multi-modal guide instruction corresponding to the independent bill sub-image, the multi-modal guide instruction being generated according to the global semantic description and a preset structured template.

[0135] The multi-modal large model calling module 920 is further configured to input the independent bill sub-image and the corresponding guide instruction into the multi-modal large model to generate invoice information structured data, the invoice information structured data being used to describe the semantic content of the independent bill sub-image.

[0136] The correction module 910 is configured to extract rigid text features of the independent bill sub-image based on an optical character recognition model, and correct the invoice information structured data based on the rigid text features.

[0137] In this way, the invoice recognition device based on the multi-modal large model provided in the embodiments of the present application generates a global semantic description of the to-be-processed invoice image through the multi-modal large model, performs regional instance segmentation on the to-be-processed invoice image by using the multi-modal large model, outputs an independent invoice sub-image corresponding to each invoice, constructs a multi-modal guiding instruction according to the global semantic description and a preset structured template, inputs the independent invoice sub-image and the corresponding guiding instruction into the multi-modal large model, and corrects invoice information structured data through rigid text features extracted by an optical character recognition model, thereby further improving the accuracy of the invoice information structured data while retaining the global reasoning advantage of the multi-modal large model.

[0138] The invoice recognition device based on the multi-modal large model provided in the embodiments of the present application can include a processor, a communication interface, a memory, and a communication bus.

[0139] The processor, the communication interface, and the memory can communicate with each other through the communication bus. The communication interface is configured to communicate with network elements such as clients or other servers. The processor is configured to execute programs, and can execute the related steps in the above-described invoice recognition method based on the multi-modal large model.

[0140] Specifically, the program can include program code including computer-executable instructions.

[0141] The processor can be a central processing unit (CPU) or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the invoice recognition device based on the multi-modal large model can be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0142] The memory is configured to store programs. The memory can include a high-speed RAM memory and can also include a non-volatile memory such as at least one disk memory.

[0143] The program can be specifically invoked by the processor to enable the invoice recognition device based on the multi-modal large model to perform the following operations:

[0144] Obtain a to-be-processed invoice image;

[0145] In a case where the invoice image to be processed is a multi-invoice splicing image, a global semantic description of the invoice image to be processed is generated by using the multi-modal large model, and region instance segmentation is performed on the invoice image to be processed by using the multi-modal large model, to output an independent invoice sub-image corresponding to each invoice;

[0146] A multi-modal guidance instruction corresponding to the independent invoice sub-image is constructed, and the multi-modal guidance instruction is generated according to the global semantic description and a preset structured template;

[0147] The independent invoice sub-image and the corresponding guidance instruction are input into the multi-modal large model to generate invoice information structured data, and the invoice information structured data is used to describe the semantic content of the independent invoice sub-image;

[0148] Rigid text features of the independent invoice sub-image are extracted based on an optical character recognition model, and the invoice information structured data is corrected based on the rigid text features.

[0149] In this way, in a case where the invoice recognition device provided in the embodiments of the present application calls a processor to execute a program, a global semantic description of the invoice image to be processed is generated by using the multi-modal large model, region instance segmentation is performed on the invoice image to be processed by using the multi-modal large model, to output an independent invoice sub-image corresponding to each invoice, and a multi-modal guidance instruction is constructed according to the global semantic description and a preset structured template, and then the independent invoice sub-image and the corresponding guidance instruction are input into the multi-modal large model, and the invoice information structured data is corrected by using rigid text features extracted by an optical character recognition model, so that the accuracy of the invoice information structured data is further improved while the global reasoning advantage of the multi-modal large model is retained.

Claims

1. An invoice recognition method based on a multimodal large model, characterized in that: include: Get the image of the invoice to be processed; In the case where the invoice image to be processed is a spliced ​​image of multiple invoices, a global semantic description of the invoice image to be processed is generated using a multimodal large model, and regional instance segmentation is performed on the invoice image to be processed using the multimodal large model to output an independent invoice sub-image corresponding to each invoice; Constructing a multimodal guidance instruction corresponding to the independent bill sub-image, wherein the multimodal guidance instruction is generated according to the global semantic description and a preset structured template; Inputting the independent bill sub-image and the corresponding guidance instruction into the multimodal macro model to generate invoice information structured data, wherein the invoice information structured data is used to describe the semantic content of the independent bill sub-image; The rigid text features of the independent bill sub-image are extracted based on an optical character recognition model, and the invoice information structured data is corrected based on the rigid text features.

2. The invoice recognition method based on a multimodal large model according to claim 1 is characterized in that: The rigid text features include digital information, a location area corresponding to the digital information, and digital confidence; The step of correcting the invoice information structured data based on the rigid text features includes: Determining corresponding comparison information in the invoice information structured data according to the location area corresponding to the digital information; In the case where the digital information is inconsistent with the comparison information and the digital confidence is greater than or equal to a first confidence threshold, overwriting the comparison information of the invoice information structured data with the digital information; When the digital information is inconsistent with the comparison information and the digital confidence is greater than or equal to the second confidence threshold and less than the first confidence threshold, the digital information is marked in the corresponding field in the invoice information structured data, and the second confidence threshold is less than the first confidence threshold.

3. The invoice recognition method based on a multimodal large model according to claim 1 is characterized in that: The rigid text features include character-level glyph features; The step of correcting the invoice information structured data based on the rigid text features includes: Arbitrating the semantic consistency between the character-level glyph features and the invoice information structured data based on a large language model; When there is semantic inconsistency between the character-level glyph features and the invoice information structured data, and the confidence of the character-level glyph features output by the large language model is greater than a third confidence threshold, the character-level glyph features are used to cover the text corresponding to the character-level glyph features in the invoice information structured data.

4. The invoice recognition method based on a multimodal large model according to claim 1 is characterized in that: The rigid text features include the spatial association relationship between each text block; The step of correcting the invoice information structured data based on the rigid text features includes: When the spatial association relationship between the two text blocks is inconsistent with the invoice information structured data, and the spatial association relationship between the two text blocks is determined to be a strong spatial association, the corresponding field mapping content in the invoice information structured data is adjusted according to the association relationship between the two text blocks.

5. The invoice recognition method based on a multimodal large model according to claim 4 is characterized in that: The spatial correlation between the two text blocks is determined to be a strong spatial correlation, specifically: The Euclidean distance between the two text blocks satisfies a preset spatial relationship condition.

6. The invoice recognition method based on a multimodal large model according to claim 5 is characterized in that: The preset spatial relationship condition is: The horizontal offset is less than a first distance threshold and the centerline vertical offset is less than a second distance threshold.

7. The invoice recognition method based on a multimodal large model according to claim 1 is characterized in that: After the step of obtaining the invoice image to be processed, the method further includes: Extracting text features of the invoice image to be processed; Identifying the language of the text features based on a language detection algorithm; According to the language, the guidance instruction template of the multimodal large model is called.

8. The invoice recognition method based on a multimodal large model according to claim 1 is characterized in that: After the step of performing region instance segmentation on the invoice image to be processed using the multimodal large model and outputting an independent bill sub-image corresponding to each bill, the method further includes: The independent bill sub-image is pre-processed, and the pre-processing includes rotation correction, boundary cropping and / or color equalization.

9. An invoice recognition device based on a multimodal large model, characterized in that: The invoice recognition device based on the multimodal large model includes: An acquisition module, used to acquire the invoice image to be processed; a multimodal large model calling module, configured to generate a global semantic description of the invoice image to be processed using the multimodal large model when the invoice image to be processed is a spliced ​​image of multiple invoices, and to perform region instance segmentation on the invoice image to be processed using the multimodal large model to output an independent invoice sub-image corresponding to each invoice; A guidance instruction construction model, used for constructing a multimodal guidance instruction corresponding to the independent bill sub-image, wherein the multimodal guidance instruction is generated according to the global semantic description and a preset structured template; The multimodal large model calling module is further used to input the independent bill sub-image and the corresponding guidance instruction into the multimodal large model to generate invoice information structured data, wherein the invoice information structured data is used to describe the semantic content of the independent bill sub-image; A correction module is used to extract rigid text features of the independent bill sub-image based on an optical character recognition model, and correct the invoice information structured data based on the rigid text features.

10. An invoice recognition device based on a multimodal large model, characterized in that: include: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the operation of the invoice recognition method based on a multimodal large model as described in any one of claims 1-8.

Images