Financial bill automatic identification generation and decision-making method and system

Abstract

The invention relates to the technical field of financial bill management, and discloses a financial bill automatic identification generation and decision-making method and system, and the method comprises the steps: firstly receiving an original bill image, carrying out the preprocessing of the original bill image, carrying out the format analysis and classification of the enhanced image through a computer vision technology, and distinguishing a known standard format from an unknown format. For a known format, accurately extracting a key region text block by applying an OCR module based on template positioning; and for an unknown format, a general document understanding model is adopted to obtain a preliminary key information key value pair. And then, the extracted information is converted into structured data through a key information structured extraction module, and the structured data is verified by a data verification and business rule engine. If the verification is passed, structured bill information is output; otherwise, sending to a manual intervention queue for further auditing. According to the method, the efficiency and accuracy of financial bill automatic processing are effectively improved, and meanwhile, the flexibility of the system for adapting to different bill formats is enhanced.

Description

Technical Field

[0001] The present application relates to the technical field of financial bill management, and more specifically, to a method and system for automatically identifying, generating and making decisions about financial bills. Background Art

[0002] In modern business operations, the processing of financial documents is a crucial component of financial management. The variety of financial documents faced by businesses is increasing, and the diversity of their formats is becoming increasingly prominent. The design and format of financial documents vary significantly across countries, regions, and industries, posing a significant challenge for automated information extraction. Traditional document processing relies primarily on manual data entry, which is inefficient, error-prone, and unable to meet the rapidly evolving needs of businesses. Therefore, how to effectively and automatically extract key information from this diverse range of document formats has become a pressing issue.

[0003] Existing technologies often face the challenge of diverse bill formats when processing financial bills. Due to their lack of adaptability to emerging or rare formats, traditional methods are typically only optimized for a few common bill formats, and are ineffective for other formats. For example, some systems use template matching to extract information. This method requires pre-defining templates for each bill format, including detailed information such as field location and type. However, faced with constantly updated and changing bill formats, maintaining these templates is time-consuming and complex, and any changes to the bill format require readjustment of the template, which greatly limits the system's flexibility and scalability.

[0004] Therefore, an optimized automatic identification, generation and decision-making solution for financial bills is expected. Summary of the Invention

[0005] In order to solve the information extraction difficulties caused by the diversity of formats faced by financial bill processing in the prior art, this application is proposed. The embodiments of this application provide a method and system for automatic identification, generation and decision making of financial bills.

[0006] In a first aspect, a method for automatic identification, generation and decision-making of financial bills is provided, comprising: receiving an original bill image uploaded by a user; performing image preprocessing on the original bill image to obtain an original bill enhanced image; performing layout analysis and classification on the original bill enhanced image based on computer vision to obtain a layout classification result; if the layout classification result is a known standard layout, inputting the original bill enhanced image into an OCR module based on template positioning to obtain a regional text block; if the layout classification result is an unknown layout, inputting the original bill enhanced image into a general document understanding model to obtain preliminary key information key-value pairs; inputting the regional text block or the preliminary key information key-value pairs into a key information structured extraction module to obtain structured bill information data; inputting the structured bill information data into a data verification and business rule engine to obtain a verification result; if the verification result is passed, outputting the structured bill information data; if the verification result is failed, sending the structured bill information data to a manual intervention queue.

[0007] In combination with the first aspect, in a possible implementation method, the original bill image is subjected to image preprocessing to obtain the original bill enhanced image, including: performing image denoising, tilt correction, brightness / contrast adjustment and image binarization on the original bill image to obtain the original bill enhanced image.

[0008] In combination with the first aspect, in a possible implementation method, the original bill enhanced image is subjected to layout analysis and classification based on computer vision to obtain a layout classification result, including: extracting local visual features and global visual features from the original bill enhanced image based on a deep convolutional neural network model to obtain a bill local visual feature coding vector and a bill global visual feature coding vector; performing feature combination on the bill local visual feature coding vector and the bill global visual feature coding vector to obtain a bill multi-scale visual feature joint coding vector; and inputting the bill multi-scale visual feature joint coding vector into a bill type multi-classifier to obtain the layout classification result.

[0009] In combination with the first aspect, in a possible implementation method, the local visual feature coding vector of the bill and the global visual feature coding vector of the bill are feature combined to obtain a joint multi-scale visual feature coding vector of the bill, including: performing information constraints on each eigenvalue of the local visual feature coding vector of the bill and each eigenvalue of the global visual feature coding vector of the bill to obtain a local visual feature coding constraint vector of the bill and a global visual feature coding constraint vector of the bill; calculating a global information domain projection operator for the local visual feature coding vector of the bill and a global information domain projection operator for the global visual feature coding vector of the bill; based on the local visual feature coding constraint vector of the bill, the global visual feature coding constraint vector of the bill, the global information domain projection operator for the local visual feature coding vector of the bill and the global information domain projection operator for the global visual feature coding vector of the bill, performing joint regularization fusion on the local visual feature coding vector of the bill and the global visual feature coding vector of the bill to obtain the joint multi-scale visual feature coding vector of the bill.

[0010] In combination with the first aspect, in a possible implementation, if the layout classification result is a known standard layout, the original bill enhanced image is input into an OCR module based on template positioning to obtain a regional text block, including: loading a template file corresponding to the known standard layout, the template file including a layout identifier, a field list and a regional coordinate definition; extracting the regional coordinate definition from the template file, and extracting the image ROI area corresponding to each key information field from the original bill enhanced image based on the regional coordinate definition to obtain a set of key information field ROI images; and inputting each key information field ROI image in the set of key information field ROI images into the OCR engine respectively to obtain the regional text block.

[0011] In combination with the first aspect, in one possible implementation, the region coordinate definition is extracted from the template file, and the image ROI area corresponding to each key information field is extracted from the original bill enhanced image based on the region coordinate definition, including: performing image alignment on the original bill enhanced image so that the original bill enhanced image is consistent with the coordinate system defined by the template file.

[0012] In combination with the first aspect, in a possible implementation method, if the layout classification result is an unknown layout, the original bill enhanced image is input into a general document understanding model to obtain preliminary key information key-value pairs, including: performing full-page OCR recognition on the original bill enhanced image to obtain all text and its position information; inputting the original bill enhanced image and all text and its position information into the trained general document understanding model to obtain a text sequence to be labeled; and converting the text sequence to be labeled into the preliminary key information key-value pairs.

[0013] In the second aspect, a financial bill automatic recognition generation and decision system is provided for executing the above-mentioned financial bill automatic recognition generation and decision method, including: an original bill image receiving module for receiving the original bill image uploaded by the user; an original bill image enhancement module for performing image preprocessing on the original bill image to obtain an original bill enhanced image; an original bill layout classification module for performing layout analysis and classification on the original bill enhanced image based on computer vision to obtain a layout classification result; a regional text block generation module for inputting the original bill enhanced image into a template-based positioning OCR module to obtain a regional text block if the layout classification result is a known standard layout; preliminary key information key-value pairs A generation module is used to input the original bill enhanced image into a general document understanding model to obtain preliminary key information key-value pairs if the layout classification result is an unknown layout; a bill information extraction module is used to input the regional text block or the preliminary key information key-value pairs into a key information structured extraction module to obtain structured bill information data; a verification result generation module is used to input the structured bill information data into a data verification and business rule engine to obtain a verification result; a bill information output module is used to output the structured bill information data if the verification result is passed; a manual intervention interface module is used to send the structured bill information data to a manual intervention queue if the verification result is failed.

[0014] Compared with the existing technology, the automatic identification, generation and decision-making method and system for financial bills provided in this application first receives the original bill image and pre-processes it, and then uses computer vision technology to analyze and classify the enhanced image to distinguish between known standard layouts and unknown layouts. For known layouts, an OCR module based on template positioning is used to accurately extract key area text blocks; for unknown layouts, a general document understanding model is used to obtain preliminary key information key-value pairs. Subsequently, the extracted information is converted into structured data through a key information structured extraction module, and verified by a data verification and business rule engine. If the verification passes, the structured bill information is output; otherwise, it is sent to the manual intervention queue for further review. This method effectively improves the efficiency and accuracy of automated processing of financial bills, while enhancing the flexibility of the system to adapt to different bill formats. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 The figure illustrates a schematic flow chart of a method for automatic identification, generation and decision-making of financial bills according to an embodiment of the present application.

[0017] Figure 2 The figure illustrates a schematic flow chart of S3 in the financial bill automatic identification, generation and decision-making method according to an embodiment of the present application.

[0018] Figure 3 The figure illustrates a schematic flow chart of a preferred embodiment of S32 in the financial bill automatic identification, generation and decision-making method according to an embodiment of the present application.

[0019] Figure 4 The figure illustrates a schematic flow chart of S4 in the financial bill automatic identification, generation and decision-making method according to an embodiment of the present application.

[0020] Figure 5 The figure illustrates another schematic flow chart of S4 in the financial bill automatic identification, generation and decision-making method according to an embodiment of the present application.

[0021] Figure 6 The figure illustrates a schematic block diagram of a financial bill automatic identification, generation and decision-making system according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described herein.

[0023] Figure 1 The figure shows a schematic flow chart of the method for automatically identifying, generating and deciding financial bills according to an embodiment of the present application. Figure 1As shown, the present application provides a method for automatic identification, generation and decision-making of financial bills, including: S1, receiving an original bill image uploaded by a user; S2, performing image preprocessing on the original bill image to obtain an original bill enhanced image; S3, performing layout analysis and classification on the original bill enhanced image based on computer vision to obtain a layout classification result; S4, if the layout classification result is a known standard layout, inputting the original bill enhanced image into an OCR module based on template positioning to obtain a regional text block; if the layout classification result is an unknown layout, inputting the original bill enhanced image into a general document understanding model to obtain a preliminary key information key-value pair; S5, inputting the regional text block or the preliminary key information key-value pair into a key information structured extraction module to obtain structured bill information data; S6, inputting the structured bill information data into a data verification and business rule engine to obtain a verification result; S7, if the verification result is passed, outputting the structured bill information data; if the verification result is failed, sending the structured bill information data to a manual intervention queue.

[0024] Specifically, in step S1, the original bill image uploaded by the user is received. It should be understood that the original bill image is not only the data source for all subsequent processing, but also the decisive factor for the accuracy of the recognition result. Therefore, if the original image cannot be received and obtained, the system will not be able to complete the content understanding and structured transformation of the bill, let alone meet the core goal of automated financial processing. This move is also directly related to the needs of diversified business scenarios. Due to the complexity and diversity of bill types, formats, and sources, users can only digitize bills in real scenes by uploading original images, and use them as a unified data entry, greatly improving the flexibility and universality of the system. In addition, the user upload link is also the main means to ensure the legality, authenticity and integrity of the bill. By obtaining the original image, the system can perform subsequent authenticity verification, anti-counterfeiting detection, etc., to further meet compliance and regulatory needs.

[0025] In one specific embodiment, a dedicated upload interface or interface module is provided on the client (e.g., a PC, mobile app, or web system) to receive raw receipt images uploaded by users. Following the system's instructions, the user acquires the receipt image by selecting a locally stored receipt photo or scan, or by directly using a camera or scanner for instant capture. The client then sends the captured or selected image file to a designated receiving interface on the backend server via a network protocol such as HTTP / HTTPS, either as a file stream or in Base64 encoding.

[0026] Specifically, in step S2, the original bill image is preprocessed to obtain an enhanced image of the original bill. It should be understood that in actual business applications, bill images generated by users through various methods such as photography and scanning often contain noise pollution (including paper texture, noise in the shooting environment, indentations, etc.), and are also prone to image skew, uneven brightness or background interference, and uneven printing quality of different bills. If the unprocessed original bill image is directly input into a subsequent deep learning analysis or OCR (Optical Character Recognition) system, it is very likely to result in low information extraction accuracy, layout analysis failure, and even interruption of the entire pipeline processing link, affecting the overall system performance. Therefore, image preprocessing becomes a key step in converting the original image into a standardized input that is convenient for subsequent computer vision algorithm processing. Through systematic optimization of the original bill image, the robustness and practicality of the financial bill recognition system can be effectively improved.

[0027] In one embodiment, the original bill image is subjected to image preprocessing to obtain the original bill enhanced image, including: performing image denoising, tilt correction, brightness / contrast adjustment and image binarization on the original bill image to obtain the original bill enhanced image. First of all, image denoising is to remove various noise signals generated by shooting conditions, scanning resolution or equipment environment. Common noise types include Gaussian noise, salt and pepper noise, etc. These noises will interfere with the normal presentation of bill information and affect subsequent recognition effects. Denoising can be implemented through a variety of image processing algorithms, such as using median filtering, Gaussian filtering and other methods to effectively retain the basic structure and text information of the bill while shielding interference information. Existing technologies can also be combined with deep learning models to adaptively remove noise in complex scenarios, such as using a deep residual network for end-to-end image denoising optimization, so that the preprocessing results can minimize the impact of noise while maintaining the details of the original image.

[0028] Next, tilt correction is performed to optimize the image tilt caused by shooting angle problems or incorrect placement during bill imaging. Tilt correction is often achieved through tilt detection and transformation. The specific method can adopt a straight line detection strategy based on Hough transform to analyze the main direction angles of the bill edge or text baseline to obtain the tilt angle. The system performs affine or perspective transformation on the image according to the tilt angle, returning the bill layout to the standard coordinate system and ensuring the pixel coordinate consistency of the key fields and the standard template. Accurate tilt correction is not only beneficial for subsequent template positioning, but also helps to improve the accuracy of the text order and spatial structure derived by OCR recognition.

[0029] The brightness and contrast adjustments in the pre-processing process mainly solve the imaging problems caused by uneven lighting in the shooting environment and unbalanced exposure of the equipment. Due to external factors such as backlight, shadows, and reflections, some bills have abnormal overall grayscale distribution or are locally too dark or too bright. At this time, brightness normalization and contrast stretching are required. Through histogram equalization, contrast adaptive enhancement and other methods, the grayscale difference between text and background can be made more obvious, thereby greatly improving the subsequent text area segmentation and character resolution capabilities. Some advanced implementation solutions will also adaptively adjust the parameters according to the local characteristics of the bill area to achieve dynamic optimization that emphasizes both global and local aspects, further enhancing the pre-processing effect.

[0030] Image binarization converts the form image into black and white, creating a clear contrast between key text and the background, thus laying a solid foundation for character segmentation and recognition in the OCR engine. Adaptive thresholding algorithms, such as Otsu's method or local adaptive thresholding, are typically used to flexibly select the optimal threshold for different regions of the document image, maximizing both text connectivity and background noise reduction. Binarization eliminates background color noise and suppresses information interference caused by complex shading or printing defects, resulting in clearer character outlines and significantly improving the accuracy and robustness of subsequent document structuring.

[0031] Specifically, in step S3, the original enhanced bill image is subjected to layout analysis and classification based on computer vision to obtain a layout classification result. It should be understood that due to the significant differences in layout design, field structure, element layout, and other aspects of various bill types, only by accurately identifying the layout can the appropriate information extraction strategy be selected, thereby effectively improving the system's compatibility, processing efficiency, and data accuracy. Against the backdrop of the continuous development of modern intelligent image processing technology, traditional rule-based or shallow image feature-based methods have been unable to meet the high requirements for accuracy, speed, and compatibility in the field of automatic bill recognition. Therefore, the original enhanced bill image is subjected to layout analysis and classification based on computer vision to obtain a layout classification result.

[0032] In one embodiment, Figure 2 As shown, the original bill enhanced image is subjected to layout analysis and classification based on computer vision to obtain a layout classification result, including: S31, extracting local visual features and global visual features from the original bill enhanced image based on a deep convolutional neural network model to obtain a bill local visual feature coding vector and a bill global visual feature coding vector; S32, performing feature combination on the bill local visual feature coding vector and the bill global visual feature coding vector to obtain a bill multi-scale visual feature joint coding vector; S33, inputting the bill multi-scale visual feature joint coding vector into a bill type multi-classifier to obtain the layout classification result.

[0033] Specifically, a deep convolutional neural network model is used to automatically extract local and global visual features. Local visual features often refer to fine-grained features such as pixel combination patterns, edges, corners, and even character textures within a specific small area of an image. These features often carry key semantic information such as the form area, digital amount area, and invoice number area. Global visual features are expressed from the macroscopic spatial structure and layout pattern of the entire image, such as the overall layout structure of the invoice, the row and column distribution of the table network, and the relative spatial relationship between the main and subtitles. This feature extraction method that gives equal weight to both local and global features can effectively alleviate the expression limitations brought about by single-scale features and enhance the model's sensitivity to subtle changes in layout and large-scale structural differences.

[0034] Common local feature extraction methods include multi-layer convolution operations and the progression of local receptive fields, which can meticulously describe the microscopic style changes of bills. For example, classic CNNs like ResNet and VGG typically extract local features such as edge textures at the bottom layer, while subsequent layers gradually stack and abstract to form a complex global structural pattern representation.

[0035] However, in real-world bill scenarios, it's difficult to accurately capture all bill style variations using only single-scale local or global features. Therefore, the local and global visual feature encoding vectors are combined to generate a joint multi-scale visual feature encoding vector. In one embodiment, the local and global visual feature encoding vectors can be combined to generate a joint multi-scale visual feature encoding vector using feature concatenation or weighted summation.

[0036] It can be seen that the layout classification result of the multi-scale visual feature joint encoding vector of the bill will directly determine the extraction of structured bill information based on a known layout or an unknown layout. Therefore, when performing feature combination of the local visual feature encoding vector of the bill and the global visual feature encoding vector of the bill, it is also necessary to consider the comprehensibility of the combined local-global visual features for the local-global structured semantic expression of the combined visual features, thereby improving the accuracy of the layout classification results.

[0037] Therefore, if Figure 3As shown, in a preferred embodiment, the local visual feature coding vector of the bill and the global visual feature coding vector of the bill are feature combined to obtain a multi-scale visual feature joint coding vector of the bill, including: S321, each eigenvalue of the local visual feature coding vector of the bill and each eigenvalue of the global visual feature coding vector of the bill are subjected to information constraints to obtain a local visual feature coding constraint vector of the bill and a global visual feature coding constraint vector of the bill, that is, for the local-global structured features of the predetermined format of the financial bill, the eigenvalue granularity structure attribute is first subjected to information constraint modeling, that is, for the local visual feature coding vector of the bill Each eigenvalue of and the bill global visual feature encoding vector Each eigenvalue of ,calculate: ;in, ,and , Represents the local visual feature encoding constraint vector of the bill Each eigenvalue of Represents the global visual feature encoding constraint vector of the bill Each eigenvalue of .

[0038] S322, calculating a global information domain projection operator for the bill local visual feature coding vector and a global information domain projection operator for the bill global visual feature coding vector, expressed as: ;in, represents the global information domain projection operator for the local visual feature encoding vector of the bill, Represents a global information domain projection operator for the global visual feature encoding vector of the bill.

[0039] That is, based on the global decomposition modeling of the information domain, joint representation space projection mapping is further performed at the global scale level.

[0040] S323: Based on the bill local visual feature encoding constraint vector, the bill global visual feature encoding constraint vector, the global information domain projection operator for the bill local visual feature encoding vector, and the global information domain projection operator for the bill global visual feature encoding vector, the bill local visual feature encoding vector and the bill global visual feature encoding vector are jointly regularized and fused to obtain the bill multi-scale visual feature joint encoding vector, which is expressed as: ;in is a predetermined feature combination function of the bill local visual feature encoding vector and the bill global visual feature encoding vector, such as feature splicing or weighted summation, and is the bias modulation vector, which can be obtained through training, It means adding by position. Represents the joint encoding vector of the multi-scale visual features of the bill.

[0041] In this way, while retaining the fine-grained information structure characteristics, the expression limitations of the local-global single scale are solved. It can be understood as representing different structural characteristics in the information domain for fine-grained details such as the text area of ​​the financial document and global scale details such as the table structure, thereby improving the accuracy of the layout classification results by improving the comprehensibility of the information structure.

[0042] Next, the resulting multi-scale visual feature joint encoding vector is fed into a multi-classifier for bill type classification, completing the classification of various bill formats. This classifier is typically based on a fully connected neural network or a more advanced weak ensemble classifier algorithm (such as LightGBM or XGBoost). By probabilistically normalizing the multi-scale joint features, it outputs the posterior probability of each layout type, ultimately determining the classification label that best approximates the bill's actual category. This step can be understood as locating the input bill within a multi-category catalog of known or unknown layouts and serves as a key criterion for selecting subsequent templated OCR parsing or general document understanding models.

[0043] Specifically, in step S4, if the layout classification result is a known standard layout, the original bill enhanced image is input into the template-based positioning OCR module to obtain a regional text block. If the layout classification result is an unknown layout, the original bill enhanced image is input into the general document understanding model to obtain preliminary key information key-value pairs. It should be understood that in the actual financial bill recognition and processing system, the diversity of layout structures and the complexity of bill sources are extremely high. The bill structure of the known standard layout generally has high stability, and the field layout, font, and relative position remain basically unchanged. Non-standard and unknown layout bills are very likely to have uncertain structures, changeable or newly added fields due to historical accumulation, localization differences or business process redundancy, so a single extraction mode is difficult to be fully applicable. For the two types of scenarios, targeted technical routes are adopted to effectively improve the accuracy, flexibility and universality of the entire automated recognition solution.

[0044] In one embodiment, Figure 4As shown, if the layout classification result is a known standard layout, the original bill enhanced image is input into the template-based positioning OCR module to obtain the regional text block, including: S41, loading a template file corresponding to the known standard layout, the template file including a layout identifier, a field list and a regional coordinate definition; S42, extracting the regional coordinate definition from the template file, and extracting the image ROI area corresponding to each key information field from the original bill enhanced image based on the regional coordinate definition to obtain a set of key information field ROI images; S43, inputting each key information field ROI image in the set of key information field ROI images into the OCR engine respectively to obtain the regional text block.

[0045] In one embodiment, the region coordinate definition is extracted from the template file, and the image ROI region corresponding to each key information field is extracted from the original bill enhanced image based on the region coordinate definition, including: performing image alignment on the original bill enhanced image so that the original bill enhanced image is consistent with the coordinate system defined by the template file.

[0046] Specifically, for known standard formats, the template-based OCR information extraction model offers significant efficiency and accuracy. This model relies on a pre-established library of template files, each of which precisely records the format identifier, logical names, and region coordinates of all key information fields for that particular invoice format. Implementation begins by selecting a template and aligning the original enhanced invoice image with the template coordinate system using an image alignment algorithm. This can be achieved using methods such as feature point registration (SIFT, SURF), template matching, or affine or perspective transformations to eliminate mapping errors caused by camera angle, scaling, and rotation. Once alignment is successful, the system directly extracts the ROI (Region of Interest) for each key field on the enhanced image based on the coordinates defined in the template. This ROI closely corresponds to the actual key information, such as the invoice code, invoice date, and amount. These ROIs are then fed into a general-purpose or specialized OCR engine (such as Tesseract, PaddleOCR, or ABBYY) for fine-grained text recognition in the text-containing regions, resulting in structured regional text blocks. This template-driven model is suitable for standard, widely used invoices such as electronic invoices, general and special VAT invoices, receipts, and motor vehicle invoices. Its advantages include eliminating the need for full-image text detection and recognition, reducing image noise by cropping targeted areas, and often outperforming full-page OCR or deep learning extraction methods for clearly printed, well-structured standard invoices. Furthermore, the flexible template rules facilitate the maintenance and expansion of new templates, even for small-scale style updates to standard invoices. Furthermore, the precise spatial definition of the template file facilitates the further development of the business rule engine's verification capabilities, reducing the risk of misidentification.

[0047] In one embodiment, Figure 5 As shown, if the layout classification result is an unknown layout, the original bill enhanced image is input into the general document understanding model to obtain preliminary key information key-value pairs, including: S44, performing full-page OCR recognition on the original bill enhanced image to obtain all texts and their position information; S45, inputting the original bill enhanced image and all texts and their position information into the trained general document understanding model to obtain a text sequence to be labeled; S46, converting the text sequence to be labeled into the preliminary key information key-value pairs.

[0048] Understandably, for unknown-format invoices, template localization methods lack prior structural support and cannot predefine the area of each field. Forced matching is not only inefficient but also prone to field position drift, mismatches, and extraction errors. Such scenarios require a general document understanding model that leverages modern deep learning, natural language processing (NLP), and visual multimodal inference capabilities to learn localization and semantic matching rules from unstructured text. This implementation involves first performing OCR on the entire image to obtain spatial descriptions, including the content and coordinates of all text lines. This text, along with positional information and multi-scale image features, is then fed into a well-trained general document understanding model. Typical models include, but are not limited to, LayoutLM, Donut, StructText, and DocFormer. These models combine spatial layout encoding, contextual semantics, and visual features to identify the "semantic role" of each line of text and infer its actual business field meaning. For example, they can automatically analyze the specific date information next to the "invoice date" text using attention mechanisms and relational modeling. The model ultimately outputs a structured text sequence (i.e., the text sequence to be labeled), which is then converted into key-value pairs through rule-based or deep learning post-processing. For example, for the "total amount" that appears in an irregular position on the bill, the model can automatically infer that the number to the right of the "total amount" text is the total amount of the bill based on the training data, and output the structured pair {"total amount": "123.00"}.

[0049] Specifically, in step S5, the regional text block or the preliminary key information key-value pair is input into the key information structured extraction module to obtain structured bill information data. It should be understood that the regional text block outputs only the text content and the layout location or logical area of its source. This text is often presented as a number of OCR recognition units (characters, words, or phrases). While the preliminary key information key-value pair achieves a certain degree of field value mapping, it has not undergone deep semantic verification and integration with business rules, and its accuracy, completeness, and usability do not meet production-level requirements. Directly applying these two methods to downstream business processes will lead to unclear structures, high error rates, increased manual intervention costs, and significantly weakened automation efficiency. Therefore, it is necessary to further utilize a structured extraction module to accommodate complex field entity relationships, cross-regional information aggregation, business context verification, regular expression verification, and multi-field combination parsing, thereby sublimating the raw text into standardized, compliant, relatable, and recordable data entities. For example, in VAT invoice recognition, core fields like "Invoice Date," "Seller Name," "Price and Tax Total," and "Verification Code" must not only maintain stable position but also meet specific structural validation and context-based mutual exclusion rules. The structured extraction module can analyze the logical loops within invoice content, ensuring, for example, that uppercase and lowercase amounts correspond, and that invoice codes and numbers have valid lengths. For some fields, missing values must be automatically inferred based on the specific format of the invoice.

[0050] In a specific embodiment, for regional text block input, the structured extraction module integrates text matching and regular rule engines, combined with a natural language processing (NLP) basic model. For example, through a template library or domain knowledge graph, the keywords, patterns, and verification logic of various key information are preset (such as the specific length of the invoice number, starting with a number, and only allowing a combination of letters and numbers). The text of each region identified by OCR is first grouped and classified, and then through technical means such as keyword neighbor search, regular expression extraction, and fuzzy matching, the text content is mapped into a fixed structure field. At the same time, common abnormal scenarios are identified (such as incorrect number separation in the amount field, non-compliant date format, incorrect ID number check digit, etc.). Taking a general value-added tax invoice as an example, after identifying the text in the "Total Price and Tax" area, the system needs to automatically extract the corresponding amount number and further verify its consistency with the content of the "Amount in Lowercase" column, and finally generate a standardized key-value pair that meets the requirements of upstream and downstream systems.

[0051] For input using preliminary key-value pairs, the implementation of the structured extraction module focuses on candidate field disambiguation, entity merging, and contextual relevance enhancement. This often relies on deep algorithms such as entity recognition models (such as LSTM-CRF with contextual weights, or upstream and downstream fine-tuning of BERT-like pre-trained models), relation extraction, and custom domain-specific business rule engines. This module performs secondary screening of uncertain fields in the model output. For example, when multiple candidate entities exist within the same text, it intelligently selects the optimal entity or merges entities based on context, field-space relationships, or historical data distribution. Furthermore, for missing or anomalous data, business rules are used to infer, dynamically correct, or complete missing fields. For example, in a motor vehicle sales invoice, if multiple suspected "Vehicle Frame Number" fields appear in the preliminary key-value pairs, the system combines a synonym library for "Vehicle Identification Code," contextual header similarity, and table position operators to extract a single valid standard field, avoiding duplication or omission.

[0052] Specifically, in step S6, the structured bill information data is input into the data verification and business rule engine to obtain the verification result. It should be understood that the structured bill information data is essentially a standardized data carrier obtained after multi-level processing such as OCR recognition, layout classification, and semantic structuring. However, due to the inevitable accumulation of errors in the technology, the complexity and variability of bill information, and the multi-dimensional superposition of actual business rules, the data obtained by the previous processing alone cannot meet the high standards and strict requirements for data quality and reliability in various business scenarios such as finance, compliance, accounting, and risk control. Therefore, a special data verification and business rule engine must be set up to conduct comprehensive and systematic verification and decision-making on the structured results, so as to achieve data credibility, traceability of results, and ensure the orderly operation of downstream application links.

[0053] Specifically, the implementation of data validation and business rule engines typically adopts a modular and strategic design approach. Taking mature product systems such as BPMN (Business Process Modeling and Notation), Drools (open source rule engine), or the Alibaba Cloud Rule Engine as examples, validation engine construction generally covers the following levels. First, basic format validation relies on the rule engine's built-in regular expressions, field type templates, and length and value range controls to verify the legitimacy of each structured field in the bill. For example, the invoice code field should be a highly formatted string with clearly defined length, character type, and value range. The date field must be formatted as "YYYY-MM-DD" and fall within the permitted publishing range. The amount field cannot contain negative values or unusual characters. This process is modeled as a rule node, allowing for flexible maintenance and expansion, and efficient execution.

[0054] Secondly, business logic validation is even more crucial in the verification process. This level of validation is implemented within the engine's business rule library, which is maintained by domain experts and covers key business scenarios and high-risk areas. For example, the "uppercase amount" and "lowercase amount" extracted from structured data must be absolutely consistent and conform to RMB writing conventions; the "verification code" field must automatically verify legitimacy based on fields such as the "invoice code" and "invoice number" combined with a verification code generation algorithm; the "invoice date" must not be later than the "current date"; and the "total amount" of each item on an invoice must be equal to the sum of the individual amounts to prevent tampering. For complex business scenarios, the rule engine supports advanced relationship validation between multiple fields. For example, certain types of special invoices may only allow for specific tax rates, and the license plate and engine number fields on motor vehicle invoices must all be automatically verified along with all license plate-related business logic. Some validation even involves invoicing external interfaces, such as connecting to the State Administration of Taxation's invoice verification platform API to perform online verification of invoice uniqueness, cancellation status, and other aspects to ensure the invoice is authentic, valid, and not duplicated.

[0055] The third implementation is to combine multiple verification modes to support dynamic policy configuration and visual business management. A mature verification and business rule engine will come with rule priority, conditional judgment and dependency control. For example, rule-based reasoning engines such as Drools support high concurrency and module independence of data verification flows through "if-then" rule sets, multi-layer nesting and conditional chains. Business users can customize new rules, enable or disable some rule nodes without changing the main program, so as to achieve continuous evolution and agile response of the system. At the same time, penalty mechanisms and fault-tolerant mechanisms are also indispensable components of the system. When multiple business rule checks fail, the system will automatically mark the associated structured data as "failed" and push it to the manual review queue. It can also perform statistical analysis on abnormal data types and frequencies, and provide feedback for front-end model optimization and business avoidance decisions.

[0056] In a specific embodiment, during the construction of the Drools rule engine, all structured invoice information data is first input into the rule engine as a Java object or a JSON carrier. The system then loads each business verification rule in the rule library. For example, if "the invoice code requires 12 digits," the rule is expressed as "when (invoice code.length == 12) then (verification passed)," and all fields are traversed and judged in sequence. For cross-field verification with complex logic, the rule engine supports joint conditions and multi-path decision-making, such as "the uppercase amount and the lowercase amount must have the same numerical value, and an exception is marked when the case correspondence algorithm verification fails." In addition, the Drools rule engine can also set the fact object (Fact) and the working memory (Working Memory) to facilitate the parallel and efficient processing of batch invoices, such as uniqueness checks within batch groups and cumulative merging logic.

[0057] Specifically, in step S7, if the verification result is passed, the structured bill information data is output; if the verification result is failed, the structured bill information data is sent to the manual intervention queue. Specifically, for a large number of standardized bills that appear frequently, their structural patterns, field formats and business logic are extremely standardized. After sufficient training, the automation engine can accurately output structured data in most cases. Directly outputting the passed data not only achieves end-to-end automation and reduces manual intervention, but also significantly improves business flow speed and operational efficiency. For example, downstream systems such as enterprise automated archiving, financial system batch accounting, online tax declaration, and automatic accounting can seamlessly integrate automatic recognition results to efficiently realize digital data flow and automatic decision-making.

[0058] In actual business, even OCR and structured engines that have undergone multiple rounds of optimization cannot guarantee data integrity and standardization when faced with atypical bills, scarce coding samples, abnormal filling, format drift, damaged bills, field ambiguity, etc. Directly outputting data in this situation will inevitably lead to data risks and may even cause significant errors in final accounting, tax reporting, compliance audits, etc. For this reason, when data verification fails, the structured information and related context of the bill must be pushed to a dedicated manual intervention queue, where experienced manual reviewers will conduct a secondary review, error correction, and supplementation to ensure that the data ultimately delivered to the main business system is sufficiently accurate and traceable, achieving efficient cooperation between machines and humans. On the one hand, the manual intervention queue blocks the automatic flow of suspicious or abnormal data, and on the other hand, it can continuously accumulate difficult scenarios and failed samples, providing high-value annotation and feedback sources for subsequent model retraining and rule base upgrades, accelerating the system's self-evolution capabilities.

[0059] In summary, the automatic recognition, generation and decision-making method for financial bills provided in this application first receives the original bill image and pre-processes it, and then uses computer vision technology to perform layout analysis and classification on the enhanced image to distinguish between known standard layouts and unknown layouts. For known layouts, an OCR module based on template positioning is used to accurately extract key area text blocks; for unknown layouts, a general document understanding model is used to obtain preliminary key information key-value pairs. Subsequently, the extracted information is converted into structured data through a key information structured extraction module, and verified by a data verification and business rule engine. If the verification passes, the structured bill information is output; otherwise, it is sent to the manual intervention queue for further review. This method effectively improves the efficiency and accuracy of automated processing of financial bills, while enhancing the flexibility of the system to adapt to different bill formats.

[0060] This application also provides a financial bill automatic identification generation and decision system, such as Figure 6As shown, the financial bill automatic recognition generation and decision system 100 includes: an original bill image receiving module 110 for receiving the original bill image uploaded by the user; an original bill image enhancement module 120 for performing image preprocessing on the original bill image to obtain an original bill enhanced image; an original bill layout classification module 130 for performing layout analysis and classification on the original bill enhanced image based on computer vision to obtain a layout classification result; a regional text block generation module 140 for inputting the original bill enhanced image into an OCR module based on template positioning to obtain a regional text block if the layout classification result is a known standard layout; a preliminary key information key-value pair generation module 150 for If the layout classification result is unknown, the original enhanced bill image is input into a general document understanding model to obtain preliminary key information key-value pairs. The bill information extraction module 160 is configured to input the regional text block or the preliminary key information key-value pairs into a key information structured extraction module to obtain structured bill information data. The verification result generation module 170 is configured to input the structured bill information data into a data verification and business rule engine to obtain a verification result. The bill information output module 180 is configured to output the structured bill information data if the verification result is passed. The manual intervention interface module 190 is configured to send the structured bill information data to a manual intervention queue if the verification result is failed.

[0061] The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this application are merely illustrative and not restrictive, and it should not be assumed that these advantages, strengths, and effects are required of each embodiment of this application. In addition, the specific details disclosed above are merely illustrative and facilitating understanding, and are not restrictive. The above details do not limit this application to necessarily being implemented using the above specific details.

[0062] The block diagrams of the devices, devices, equipment, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0063] It should also be noted that in the apparatus, device, and method of the present application, each component or each step can be decomposed and / or recombined, and such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0064] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0065] The above description has been provided for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for automatically identifying, generating and deciding financial bills, characterized in that: include: Receiving the original bill image uploaded by the user; Performing image preprocessing on the original bill image to obtain an original bill enhanced image; Performing layout analysis and classification on the original bill enhanced image based on computer vision to obtain a layout classification result; If the format classification result is a known standard format, inputting the original bill enhanced image into an OCR module based on template positioning to obtain a regional text block; If the layout classification result is an unknown layout, inputting the original bill enhanced image into a general document understanding model to obtain preliminary key information key-value pairs; Inputting the regional text block or the preliminary key information key-value pair into a key information structured extraction module to obtain structured bill information data; Inputting the structured bill information data into a data verification and business rule engine to obtain a verification result; If the verification result is passed, the structured bill information data is output; if the verification result is failed, the structured bill information data is sent to the manual intervention queue.

2. The method for automatic identification, generation and decision-making of financial bills according to claim 1, characterized in that: Performing image preprocessing on the original bill image to obtain an original bill enhanced image includes: The original bill image is subjected to image denoising, tilt correction, brightness / contrast adjustment and image binarization to obtain the original bill enhanced image.

3. The method for automatic identification, generation and decision-making of financial bills according to claim 1, characterized in that: Performing layout analysis and classification on the original enhanced bill image based on computer vision to obtain a layout classification result includes: Extracting local visual features and global visual features from the original bill enhanced image based on a deep convolutional neural network model to obtain a bill local visual feature encoding vector and a bill global visual feature encoding vector; Combining the local visual feature coding vector of the bill and the global visual feature coding vector of the bill to obtain a joint multi-scale visual feature coding vector of the bill; The multi-scale visual feature joint encoding vector of the bill is input into the bill type multi-classifier to obtain the format classification result.

4. The method for automatic identification, generation and decision-making of financial bills according to claim 1, characterized in that: Combining the local visual feature coding vector of the bill and the global visual feature coding vector of the bill to obtain a joint multi-scale visual feature coding vector of the bill includes: Performing information constraints on each eigenvalue of the bill local visual feature coding vector and each eigenvalue of the bill global visual feature coding vector to obtain a bill local visual feature coding constraint vector and a bill global visual feature coding constraint vector; Calculating a global information domain projection operator for the local visual feature coding vector of the bill and a global information domain projection operator for the global visual feature coding vector of the bill; Based on the local visual feature coding constraint vector of the bill, the global visual feature coding constraint vector of the bill, the global information domain projection operator for the local visual feature coding vector of the bill, and the global information domain projection operator for the global visual feature coding vector of the bill, the local visual feature coding vector of the bill and the global visual feature coding vector of the bill are jointly regularized and fused to obtain the multi-scale visual feature joint coding vector of the bill.

5. The method for automatic identification, generation and decision-making of financial bills according to claim 1, characterized in that: If the format classification result is a known standard format, the original enhanced bill image is input into an OCR module based on template positioning to obtain a regional text block, including: Loading a template file corresponding to the known standard layout, the template file including a layout identifier, a field list, and an area coordinate definition; Extracting a region coordinate definition from the template file, and extracting an image ROI region corresponding to each key information field from the original bill enhanced image based on the region coordinate definition to obtain a set of key information field ROI images; Each key information field ROI image in the set of key information field ROI images is input into an OCR engine to obtain the regional text block.

6. The method for automatic identification, generation and decision-making of financial bills according to claim 5, characterized in that: Extracting a region coordinate definition from the template file, and extracting an image ROI region corresponding to each key information field from the original bill enhanced image based on the region coordinate definition, including: Perform image alignment on the original bill enhanced image so that the original bill enhanced image is consistent with the coordinate system defined by the template file.

7. The method for automatic identification, generation and decision-making of financial bills according to claim 1, characterized in that: If the layout classification result is an unknown layout, the original enhanced bill image is input into a general document understanding model to obtain preliminary key information key-value pairs, including: Performing full-page OCR recognition on the original enhanced image of the bill to obtain all text and its position information; Inputting the original enhanced image of the bill and all text and its position information into the trained general document understanding model to obtain a text sequence to be labeled; The text sequence to be labeled is converted into the preliminary key information key-value pairs.

8. A financial bill automatic identification, generation and decision-making system, used to execute the financial bill automatic identification, generation and decision-making method according to any one of claims 1 to 7, characterized in that: include: The original bill image receiving module is used to receive the original bill image uploaded by the user; An original bill image enhancement module, configured to perform image preprocessing on the original bill image to obtain an original bill enhanced image; An original bill layout classification module, configured to perform layout analysis and classification on the original bill enhanced image based on computer vision to obtain a layout classification result; A regional text block generating module is configured to input the original bill enhanced image into an OCR module based on template positioning to obtain a regional text block if the layout classification result is a known standard layout; a preliminary key information key-value pair generation module, configured to input the original bill enhanced image into a general document understanding model to obtain preliminary key information key-value pairs if the layout classification result is an unknown layout; a bill information extraction module, configured to input the regional text block or the preliminary key information key-value pair into a key information structured extraction module to obtain structured bill information data; A verification result generating module, configured to input the structured bill information data into a data verification and business rule engine to obtain a verification result; A bill information output module, configured to output the structured bill information data if the verification result is passed; The manual intervention interface module is used to send the structured bill information data to the manual intervention queue if the verification result is failure.

Images