A method and system for automatic identification, generation, and decision-making of financial documents

Abstract

This application relates to the field of financial document management technology, and discloses a method and system for automatic identification, generation, and decision-making of financial documents. The method first receives and preprocesses the original document image, then uses computer vision technology to perform layout analysis and classification on the enhanced image, distinguishing between known standard layouts and unknown layouts. For known layouts, a template-based OCR module is used to accurately extract key text blocks; for unknown layouts, a general document understanding model is used to obtain preliminary key-value pairs of key information. Subsequently, the extracted information is transformed into structured data through a key information structure extraction module, and verified by a data validation and business rule engine. If the validation passes, the structured document information is output; otherwise, it is sent to a manual intervention queue for further review. This method effectively improves the efficiency and accuracy of automated financial document processing, while enhancing the system's flexibility in adapting to different document layouts.

Description

Technical Field

[0001] This application relates to the field of financial document management technology, and more specifically, to a method and system for automatic identification, generation, and decision-making of financial documents. Background Technology

[0002] In modern enterprise operations, the processing of financial documents is a crucial component of financial management. Enterprises are facing an increasing variety of financial documents, with a particularly pronounced diversity in formats. Significant differences in the design and format of financial documents exist across different countries, regions, and industries, posing a substantial challenge to automated information extraction. Traditional document processing methods rely primarily on manual data entry, which is not only inefficient and error-prone but also ill-suited to the rapidly changing needs of businesses. Therefore, effectively and accurately extracting key information automatically from diverse document formats has become a pressing issue.

[0003] Existing technologies often face the challenge of diverse document formats when processing financial documents. Lacking the ability to adapt to newly emerging or less common formats, traditional methods are typically optimized only for a few fixed, common document formats, while performing poorly on other formats. For example, some systems use template matching for information extraction, which requires pre-defining a template for each document format, including detailed information such as field positions and types. However, maintaining these templates is time-consuming and complex in the face of constantly updated and changing document formats, and requires template readjustment whenever the document format changes, severely limiting the system's flexibility and scalability.

[0004] Therefore, an optimized scheme for automatic identification, generation, and decision-making of financial documents is needed. Summary of the Invention

[0005] This application addresses the difficulties in information extraction caused by the diverse formats of financial documents in existing technologies. Embodiments of this application provide a method and system for automatic identification, generation, and decision-making regarding financial documents.

[0006] In a first aspect, a method for automatic identification, generation, and decision-making of financial invoices is provided, comprising: receiving an original invoice image uploaded by a user; performing image preprocessing on the original invoice image to obtain an enhanced original invoice image; performing layout analysis and classification on the enhanced original invoice image based on computer vision to obtain a layout classification result; if the layout classification result is a known standard layout, inputting the enhanced original invoice image into a template-based OCR module to obtain a region text block; if the layout classification result is an unknown layout, inputting the enhanced original invoice image into a general document understanding model to obtain preliminary key information key-value pairs; inputting the region text block or the preliminary key information key-value pairs into a key information structure extraction module to obtain structured invoice information data; inputting the structured invoice information data into a data verification and business rule engine to obtain a verification result; if the verification result is passed, outputting the structured invoice information data; if the verification result is failed, sending the structured invoice information data to a manual intervention queue.

[0007] In conjunction with the first aspect, in one possible implementation, image preprocessing is performed on the original ticket image to obtain an enhanced original ticket image, including: image denoising, tilt correction, brightness / contrast adjustment, and image binarization of the original ticket image to obtain the enhanced original ticket image.

[0008] In conjunction with the first aspect, in one possible implementation, the original enhanced invoice image is subjected to layout analysis and classification based on computer vision to obtain a layout classification result. This includes: extracting local and global visual features from the original enhanced invoice image using a deep convolutional neural network model to obtain a local visual feature encoding vector and a global visual feature encoding vector; combining the local and global visual feature encoding vectors to obtain a multi-scale visual feature joint encoding vector; and inputting the multi-scale visual feature joint encoding vector into an invoice type multi-classifier to obtain the layout classification result.

[0009] In conjunction with the first aspect, in one possible implementation, the local visual feature encoding vector and the global visual feature encoding vector of the invoice are combined to obtain a joint encoding vector of multi-scale visual features of the invoice. This includes: applying information constraints to each feature value of the local visual feature encoding vector and each feature value of the global visual feature encoding vector of the invoice to obtain a local visual feature encoding constraint vector and a global visual feature encoding constraint vector of the invoice; calculating a global information domain projection operator for the local visual feature encoding vector and a global information domain projection operator for the global visual feature encoding vector of the invoice; and performing joint regularization fusion on the local visual feature encoding vector and the global visual feature encoding vector of the invoice based on the local visual feature encoding constraint vector, the global visual feature encoding constraint vector, the global information domain projection operator for the local visual feature encoding vector, and the global information domain projection operator for the global visual feature encoding vector of the invoice to obtain the joint encoding vector of multi-scale visual features of the invoice.

[0010] In conjunction with the first aspect, in one possible implementation, if the layout classification result is a known standard layout, the original enhanced invoice image is input into a template-based OCR module to obtain a region text block. This includes: loading a template file corresponding to the known standard layout, the template file including a layout identifier, a field list, and a region coordinate definition; extracting the region coordinate definition from the template file, and extracting the image ROI region corresponding to each key information field from the original enhanced invoice image based on the region coordinate definition to obtain a set of key information field ROI images; and inputting each key information field ROI image in the set of key information field ROI images into the OCR engine to obtain the region text block.

[0011] In conjunction with the first aspect, in one possible implementation, extracting region coordinate definitions from the template file and extracting the image ROI region corresponding to each key information field from the original invoice enhancement image based on the region coordinate definitions includes: performing image alignment on the original invoice enhancement image so that the original invoice enhancement image is consistent with the coordinate system defined by the template file.

[0012] In conjunction with the first aspect, in one possible implementation, if the format classification result is an unknown format, the original enhanced document image is input into a general document understanding model to obtain preliminary key information key-value pairs, including: performing full-page OCR recognition on the original enhanced document image to obtain all text and its location information; inputting the original enhanced document image and all text and its location information into a trained general document understanding model to obtain a text sequence to be labeled; and converting the text sequence to be labeled into the preliminary key information key-value pairs.

[0013] Secondly, an automatic financial document recognition, generation, and decision-making system is provided to execute the aforementioned automatic financial document recognition, generation, and decision-making method. This system includes: an original document image receiving module for receiving original document images uploaded by a user; an original document image enhancement module for preprocessing the original document image to obtain an enhanced original document image; an original document format classification module for performing format analysis and classification on the enhanced original document image based on computer vision to obtain a format classification result; a region text block generation module for inputting the enhanced original document image into a template-based OCR module to obtain region text blocks if the format classification result is a known standard format; and preliminary key information key-value pairs. The document includes a generation module for inputting the original enhanced document image into a general document understanding model to obtain preliminary key-value pairs if the format classification result is an unknown format; a document information extraction module for inputting the region text block or the preliminary key-value pairs into a key information structure extraction module to obtain structured document information data; a verification result generation module for inputting the structured document information data into a data verification and business rule engine to obtain a verification result; a document information output module for outputting the structured document information data if the verification result is passed; and a manual intervention interface module for sending the structured document information data to a manual intervention queue if the verification result is failed.

[0014] Compared with existing technologies, the financial invoice automatic recognition, generation, and decision-making method and system provided in this application first receives and preprocesses the original invoice image. Then, it uses computer vision technology to perform layout analysis and classification on the enhanced image, distinguishing between known standard layouts and unknown layouts. For known layouts, a template-based OCR module is applied to accurately extract key area text blocks; for unknown layouts, a general document understanding model is used to obtain preliminary key-value pairs of key information. Subsequently, the extracted information is transformed into structured data through a key information structure extraction module, and then verified by a data validation and business rule engine. If the verification passes, the structured invoice information is output; otherwise, it is sent to a manual intervention queue for further review. This method effectively improves the efficiency and accuracy of automated financial invoice processing, while enhancing the system's flexibility in adapting to different invoice layouts. Attached Figure Description

[0015] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 The illustration shows a schematic flowchart of a method for automatically identifying, generating, and making decisions on financial documents according to an embodiment of this application.

[0017] Figure 2 The illustration shows a schematic flowchart of S3 in the financial document automatic identification, generation, and decision-making method according to an embodiment of this application.

[0018] Figure 3 The illustration shows a schematic flowchart of a preferred embodiment of S32 in the financial document automatic identification, generation, and decision-making method according to an embodiment of this application.

[0019] Figure 4 The illustration shows a schematic flowchart of S4 in the financial document automatic identification, generation, and decision-making method according to an embodiment of this application.

[0020] Figure 5 The illustration shows another schematic flowchart of S4 in the financial document automatic identification, generation, and decision-making method according to an embodiment of this application.

[0021] Figure 6 The illustration shows a schematic block diagram of an automatic financial document identification, generation, and decision-making system according to an embodiment of this application. Detailed Implementation

[0022] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0023] Figure 1 The illustration shows a schematic flowchart of a method for automatically identifying, generating, and making decisions regarding financial documents according to an embodiment of this application. Figure 1As shown, this application provides a method for automatic identification, generation, and decision-making of financial invoices, including: S1, receiving an original invoice image uploaded by a user; S2, performing image preprocessing on the original invoice image to obtain an enhanced original invoice image; S3, performing layout analysis and classification on the enhanced original invoice image based on computer vision to obtain a layout classification result; S4, if the layout classification result is a known standard layout, inputting the enhanced original invoice image into a template-based OCR module to obtain a region text block; if the layout classification result is an unknown layout, inputting the enhanced original invoice image into a general document understanding model to obtain preliminary key information key-value pairs; S5, inputting the region text block or the preliminary key information key-value pairs into a key information structure extraction module to obtain structured invoice information data; S6, inputting the structured invoice information data into a data verification and business rule engine to obtain a verification result; S7, if the verification result is successful, outputting the structured invoice information data; if the verification result is unsuccessful, sending the structured invoice information data to a manual intervention queue.

[0024] Specifically, in step S1, the system receives the original invoice image uploaded by the user. It should be understood that the original invoice image is not only the data source for all subsequent processing but also a decisive factor in the accuracy of the recognition results. Therefore, if the original image cannot be received and acquired, the system will be unable to complete the understanding and structured transformation of the invoice content, and will not be able to meet the core objective of automated financial processing. This also directly relates to the needs of diverse business scenarios. Due to the complexity and diversity of invoice types, formats, and sources, users can only digitize invoices in real-world scenarios by uploading the original image, using it as a unified data entry point, greatly improving the system's flexibility and universality. Furthermore, the user upload process is also a primary means of ensuring the legality, authenticity, and integrity of invoices. By acquiring the original image, the system can perform subsequent authenticity verification, anti-counterfeiting detection, etc., further meeting compliance and regulatory requirements.

[0025] In one specific embodiment, a dedicated upload interface or module is set up on the client (e.g., PC, mobile app, or web system) to receive the original document images uploaded by the user. Following system guidance on the interface, the user acquires the document image by selecting a locally stored photo or scanned copy of the document, or by directly using a camera or scanner for real-time capture. Subsequently, the client sends the captured or selected image file to the designated receiving interface of the backend server via network protocols such as HTTP / HTTPS, in the form of a file stream or Base64 encoded data.

[0026] Specifically, in step S2, the original invoice image is preprocessed to obtain an enhanced original invoice image. It should be understood that in practical business applications, invoice images generated by users through various methods such as shooting and scanning often suffer from noise pollution (including paper texture, noise in the shooting environment, indentations, etc.), and are also prone to problems such as image skew, uneven brightness, background interference, and inconsistent printing quality among different invoices. If the unprocessed original invoice image is directly input into subsequent deep learning analysis or OCR (Optical Character Recognition) systems, it is highly likely to result in low information extraction accuracy, layout analysis failure, or even the interruption of the entire processing pipeline, affecting the overall system performance. Therefore, image preprocessing is a crucial step in converting the original image into a standardized input that is easy for subsequent computer vision algorithms to process. Through systematic optimization of the original invoice image, the robustness and practicality of the financial invoice recognition system are effectively improved.

[0027] In one embodiment, image preprocessing is performed on the original ticket image to obtain an enhanced original ticket image, including: image denoising, tilt correction, brightness / contrast adjustment, and image binarization of the original ticket image. First, image denoising aims to remove various noise signals generated by shooting conditions, scanning resolution, or device environment. Common noise types include Gaussian noise and salt-and-pepper noise, which can interfere with the normal presentation of ticket information and affect subsequent recognition. Denoising can be implemented through various image processing algorithms, such as median filtering and Gaussian filtering, to effectively preserve the basic structure and text information of the ticket while masking interference. Existing technologies can also combine deep learning models to adaptively remove noise in complex scenes, such as using deep residual networks for end-to-end image denoising optimization, so that the preprocessing result minimizes the impact of noise while maintaining the details of the original image.

[0028] Next, tilt correction optimizes the image tilt caused by shooting angle issues or improper placement during the document imaging process. Tilt correction is often achieved through tilt detection and transformation. Specifically, a line detection strategy based on Hough transform can be used to analyze the main direction angles of the document edges or text baselines to obtain the tilt angle. The system then performs affine or perspective transformations on the image based on the tilt angle, restoring the document layout to the standard coordinate system and ensuring consistency between the pixel coordinates of key fields and the standard template. Accurate tilt correction not only facilitates subsequent template positioning but also helps improve the accuracy of the text order and spatial structure exported by OCR recognition.

[0029] Brightness and contrast adjustments in the preprocessing workflow primarily address imaging issues caused by uneven lighting in the shooting environment and imbalanced equipment exposure. Some tickets, due to external factors such as backlighting, shadows, and reflections, exhibit abnormal overall grayscale distribution or areas that are too dark or too bright. In these cases, brightness normalization and contrast stretching are necessary. Histogram equalization and adaptive contrast enhancement can make the grayscale difference between text and background more pronounced, thereby significantly improving subsequent text segmentation and character recognition capabilities. Some advanced implementations also adaptively adjust parameters based on the local features of the ticket area, achieving dynamic optimization that balances global and local aspects, further enhancing the preprocessing effect.

[0030] Image binarization converts the form image into black and white, creating a sharp contrast between the key text and the background color, thus laying a solid foundation for character segmentation and recognition by the OCR engine. Adaptive thresholding algorithms, such as Otsu's method or local adaptive thresholding, are typically used to flexibly select the optimal threshold for different regions of the document image, maximizing both text connectivity and background noise reduction. Binarization eliminates background color noise and suppresses information interference caused by complex background patterns or printing defects, resulting in clearer character outlines and significantly improving the accuracy and robustness of subsequent document structuring processing.

[0031] Specifically, in step S3, the original enhanced document image is analyzed and classified using computer vision to obtain a format classification result. It should be understood that, due to significant differences in layout design, field structure, and element arrangement among various types of documents, only through accurate format identification can appropriate information extraction strategies be selected, thereby effectively improving the system's compatibility, processing efficiency, and data accuracy. With the continuous development of modern intelligent image processing technology, traditional methods based on rules or shallow image features are no longer sufficient to meet the high requirements for accuracy, speed, and compatibility in the field of automatic document recognition. Therefore, the original enhanced document image is analyzed and classified using computer vision to obtain a format classification result.

[0032] In one embodiment, such as Figure 2 As shown, the layout analysis and classification of the original enhanced invoice image based on computer vision to obtain the layout classification result includes: S31, extracting local visual features and global visual features from the original enhanced invoice image based on a deep convolutional neural network model to obtain a local visual feature encoding vector and a global visual feature encoding vector; S32, combining the local visual feature encoding vector and the global visual feature encoding vector to obtain a joint encoding vector of multi-scale visual features of the invoice; S33, inputting the joint encoding vector of multi-scale visual features of the invoice into an invoice type multi-classifier to obtain the layout classification result.

[0033] Specifically, a deep convolutional neural network model is used to automatically extract both local and global visual features. Local visual features often refer to fine-grained features such as pixel combination patterns, edges, corners, and even character textures within specific small regions of an image. These features often carry key semantic information such as form areas, numerical amount areas, and invoice number areas. Global visual features, on the other hand, express the macroscopic spatial structure and layout patterns of the entire image, such as the overall layout structure of an invoice, the row and column distribution of a table network, and the relative spatial relationship between the main and sub-titles. This feature extraction approach, which emphasizes both local and global features, can effectively alleviate the limitations of single-scale features and enhance the model's sensitivity to subtle changes in layout and large-scale structural differences.

[0034] Common local feature extraction methods include multi-layer convolutional operations and progressive local receptive fields, which can meticulously describe the microscopic style changes of documents. Taking classic CNNs such as ResNet and VGG as examples, their first few layers usually extract local features such as edge textures from the lower layers, while the later layers are gradually stacked to abstract and form a complex structural pattern representation facing the whole.

[0035] However, in real-world invoice scenarios, it is difficult to accurately cover all style variations of an invoice using only local or global features at a single scale. Therefore, the local visual feature encoding vector and the global visual feature encoding vector of the invoice are combined to obtain a joint encoding vector of multi-scale visual features. In one embodiment, the joint encoding vector of multi-scale visual features can be obtained by combining the local visual feature encoding vector and the global visual feature encoding vector of the invoice through feature concatenation or weighted summation.

[0036] As can be seen, the layout classification result of the joint encoding vector of the multi-scale visual features of the bill directly determines the extraction of structured bill information based on known or unknown layouts. Therefore, when combining the local visual feature encoding vector and the global visual feature encoding vector of the bill, it is also necessary to consider the understandability of the combined local-global visual features for the local-global structured semantic expression of the combined visual features, so as to improve the accuracy of the layout classification result.

[0037] Therefore, as Figure 3As shown, in a preferred embodiment, the local visual feature encoding vector and the global visual feature encoding vector of the bill are combined to obtain a multi-scale visual feature joint encoding vector of the bill, including: S321, applying information constraints to each feature value of the local visual feature encoding vector and each feature value of the global visual feature encoding vector of the bill to obtain a local visual feature encoding constraint vector and a global visual feature encoding constraint vector of the bill. That is, for the local-global structured features of the predetermined format of the financial bill, firstly, information constraint modeling is performed on the granular structural attributes of the feature values, i.e., for the local visual feature encoding vector of the bill... Each eigenvalue and the global visual feature encoding vector of the ticket Each eigenvalue ,calculate: ;in, ,and , Represents the constraint vector for encoding local visual features of a bill. Each feature value, Represents the global visual feature encoding constraint vector of the invoice. Each characteristic value.

[0038] S322, calculate the global information domain projection operator for the local visual feature encoding vector of the document and the global information domain projection operator for the global visual feature encoding vector of the document, expressed as: ;in, This represents the global information domain projection operator for the local visual feature encoding vector of the document. This represents the global information domain projection operator for the global visual feature encoding vector of the document.

[0039] That is, based on the global decomposition modeling of the information domain, a joint representation space projection mapping is further performed at the global scale level.

[0040] S323, based on the local visual feature encoding constraint vector of the bill, the global visual feature encoding constraint vector of the bill, the global information domain projection operator for the local visual feature encoding vector of the bill, and the global information domain projection operator for the global visual feature encoding vector of the bill, the local visual feature encoding vector of the bill and the global visual feature encoding vector of the bill are jointly regularized and fused to obtain the multi-scale visual feature joint encoding vector of the bill, expressed as: ;in It is a predetermined feature combination function of the local visual feature encoding vector and the global visual feature encoding vector of the document, such as feature concatenation or weighted summation, and It is the bias modulation vector, which can be obtained through training. This indicates addition by position. This represents the joint encoding vector of the multi-scale visual features of the ticket.

[0041] In this way, while preserving the fine-grained information structure characteristics, the limitations of local-global single-scale expression are solved. It can be understood that different structural characteristics are represented under the information domain for fine-grained details such as text areas of financial documents and global scale details such as table structure, thereby improving the accuracy of layout classification results by enhancing the comprehensibility of information structure.

[0042] Next, the obtained multi-scale visual feature joint encoding vector is imported into a multi-classifier for ticket types to complete the classification of various ticket formats. This classifier is typically based on a fully connected neural network or a more advanced weak ensemble algorithm (such as LightGBM or XGBoost). By normalizing the probabilities of the multi-scale joint features, it outputs the posterior probability of each format, ultimately determining the classification label that is closest to the actual category of the ticket. This step can be understood as locating the input ticket in a directory of known or unknown formats across multiple categories, and it is a key criterion for subsequent templated OCR parsing or selection of a general document understanding model.

[0043] Specifically, in step S4, if the format classification result is a known standard format, the original enhanced document image is input into a template-based OCR module to obtain regional text blocks. If the format classification result is an unknown format, the original enhanced document image is input into a general document understanding model to obtain preliminary key-value pairs. It should be understood that in real-world financial document recognition and processing systems, the diversity of format structures and the complexity of document sources are extremely high. Documents with known standard formats generally possess high stability, with field layouts, fonts, and relative positions remaining largely unchanged. However, non-standard or unknown formats are highly likely to have uncertain structures, volatile or newly added fields due to historical accumulation, localization differences, or redundant business processes. Therefore, a single extraction mode is difficult to apply comprehensively. For these two scenarios, targeted technical approaches have effectively improved the accuracy, flexibility, and universality of the entire automated recognition solution.

[0044] In one embodiment, such as Figure 4As shown, if the format classification result is a known standard format, the original invoice enhancement image is input into a template-based OCR module to obtain a region text block, including: S41, loading a template file corresponding to the known standard format, the template file including a format identifier, a field list, and a region coordinate definition; S42, extracting the region coordinate definition from the template file, and extracting the image ROI region corresponding to each key information field from the original invoice enhancement image based on the region coordinate definition to obtain a set of key information field ROI images; S43, inputting each key information field ROI image in the set of key information field ROI images into the OCR engine to obtain the region text block.

[0045] In one embodiment, extracting region coordinate definitions from the template file and extracting the image ROI region corresponding to each key information field from the original invoice enhancement image based on the region coordinate definitions includes: performing image alignment on the original invoice enhancement image so that the original invoice enhancement image is consistent with the coordinate system defined in the template file.

[0046] Specifically, for known standard formats, template-based OCR information extraction exhibits significant efficiency and accuracy. This model relies on a pre-established template file library, where each template file precisely records the format identifier, logical names of all key information fields, and region coordinates for that type of invoice. During implementation, the corresponding template is first selected, and the original invoice enhancement image is aligned with the template coordinate system using an image alignment algorithm. This can be achieved through methods such as feature point registration (SIFT, SURF), template matching, affine transformation, or perspective transformation, eliminating mapping errors caused by shooting angle, scaling, and rotation. After successful alignment, the system can directly extract the Region of Interest (ROI) of each key field from the enhanced image based on the coordinates defined in the template. This ROI highly corresponds to the actual key information, such as the invoice code, invoice date, and amount. These ROI regions are then input into a general or specialized OCR engine (such as Tesseract, PaddleOCR, ABBYY, etc.) for fine-grained text recognition of text-containing areas, resulting in structured text blocks. This template-driven approach is suitable for standard template documents widely used in various industries, such as electronic invoices, general / special VAT invoices, receipts, and motor vehicle invoices. Its advantages include: no need for full-image text detection and recognition; cropping and directional regions reduce image noise interference; and for clearly printed, structurally standardized documents, field recognition accuracy is often superior to full-page OCR or deep learning extraction methods. Furthermore, the template rules are flexible, facilitating maintenance and expansion with new templates, and handling minor style updates to standard documents. In addition, the precise spatial definition of the template file allows for the validation capabilities of the business rule engine to be deployed more effectively, reducing the risk of misidentification.

[0047] In one embodiment, such as Figure 5 As shown, if the format classification result is an unknown format, the original enhanced document image is input into a general document understanding model to obtain preliminary key information key-value pairs, including: S44, performing full-page OCR recognition on the original enhanced document image to obtain all text and its location information; S45, inputting the original enhanced document image and all text and its location information into the trained general document understanding model to obtain the text sequence to be labeled; S46, converting the text sequence to be labeled into the preliminary key information key-value pairs.

[0048] It is understandable that for invoices with unknown formats, template-based localization methods lack prior structural support and cannot pre-define the region of each field. Forced matching is not only inefficient but also prone to field position drift, mismatch, and extraction errors. Such scenarios require relying on general document understanding models, fully utilizing modern deep learning, NLP, and visual multimodal inference capabilities to learn localization and semantic matching rules from unstructured text. In implementation, the entire image is first read using OCR to obtain spatial descriptions such as the content and coordinates of all text lines. Then, the text and positional information, along with multi-scale image features, are input into a well-trained general document understanding model. Typical models include, but are not limited to, LayoutLM, Donut, StructText, and DocFormer. These models can combine spatial layout encoding, contextual semantics, and visual features to determine the "semantic role" of each line of text and infer its actual business field meaning. For example, based on attention mechanisms and relational modeling, they can automatically analyze the specific date information corresponding to the "invoice date" text. Finally, the model outputs a structured text sequence (i.e., the text sequence to be labeled), which, with rule-based or deep learning post-processing, is converted into key-value pairs. For example, if the "Total Amount" text appears in an irregular position on the ticket, the model can automatically infer the right-hand neighbor of the "Total Amount" text as the total amount of the ticket based on the training data, and output the structured pair {"Total Amount": "123.00"}.

[0049] Specifically, in step S5, the regional text block or the preliminary key information key-value pair is input into the key information structure extraction module to obtain structured invoice information data. It should be understood that the regional text block only outputs the text content and its source page location or logical area; this text is often presented as several OCR recognition units (characters, words, or phrases). Although the preliminary key information key-value pair achieves a certain degree of field value mapping, it has not undergone deep semantic verification and integration with business rules, and its accuracy, completeness, and usability do not yet meet production-grade requirements. Directly using these two in downstream business processes will lead to unclear structure, high error rates, increased manual intervention costs, and significantly weaken automation efficiency. Therefore, it is necessary to further utilize the structure extraction module to accommodate complex field entity relationships, cross-regional information aggregation, business context verification, regular expression verification, and multi-field combination parsing, transforming the original text into standardized, compliant, associative, and recordable data entities. For example, in VAT invoice recognition, core fields such as "Invoice Date," "Seller Name," "Total Price (including tax)," and "Verification Code" must not only remain in their fixed positions, but their content must also meet specific structural validation and contextual exclusion rules. The structured extraction module can parse the logical loops between invoice content, such as ensuring a one-to-one correspondence between uppercase and lowercase amounts, valid invoice codes and numbers, and automatically inferring missing values ​​for certain fields based on the specific format of the invoice.

[0050] In one specific embodiment, for input text blocks, the structured extraction module integrates a text matching and regular expression engine, combined with a basic Natural Language Processing (NLP) model. For example, using a template library or domain knowledge graph, it presets keywords, patterns, and verification logic for various key information (such as a specific length for invoice numbers, starting with numbers, and allowing only combinations of letters and numbers). The OCR-recognized text blocks are first grouped and categorized, and then, through techniques such as keyword nearest neighbor search, regular expression extraction, and fuzzy matching, the text content is mapped to fixed structure fields. Simultaneously, common anomalies are identified (such as incorrect number separation in the amount field, non-compliant date formats, and incorrect ID number verification digits). Taking a general VAT invoice as an example, after recognizing the text in the "total price and tax" area, the system automatically extracts the corresponding amount and further verifies its consistency with the content in the "amount in lowercase" column, ultimately generating standardized key-value pairs that meet the requirements of upstream and downstream systems.

[0051] For scenarios where initial key-value pairs are used as input, the structured extraction module focuses on candidate field disambiguation, entity merging, and enhanced contextual association. This often relies on deep learning algorithms such as entity recognition models (e.g., LSTM-CRF with context weights, or upstream and downstream fine-tuning of BERT-like pre-trained models), relation extraction, and custom domain business rule engines. On one hand, it performs secondary screening of uncertain fields in the model output. For example, when multiple candidate entities exist in a single text, it intelligently judges based on context, relative relationships in the field space, or historical data distribution to select the optimal entity or complete entity merging. On the other hand, for missing or abnormal data, it uses business rules to infer, dynamically correct, or complete missing fields. For instance, in a motor vehicle sales invoice, if multiple suspected "Vehicle Identification Number" fields appear in the initial key-value pairs, the system combines a "Vehicle Identification Code" synonym library, context header similarity, and table position operators to extract a unique and valid standard field, avoiding duplication or omission.

[0052] Specifically, in step S6, the structured invoice information data is input into the data verification and business rule engine to obtain the verification result. It should be understood that structured invoice information data is essentially a standardized data carrier obtained after multiple levels of processing, including OCR recognition, format classification, and semantic structuring. However, due to the unavoidable accumulation of technical errors, the complexity and variability of invoice information, and the diverse superposition of actual business rules, the data obtained from the preceding processing alone cannot meet the high standards and strict requirements for data quality and reliability in various business scenarios such as finance, compliance, accounting, and risk control. Therefore, it is necessary to set up a dedicated data verification and business rule engine to conduct comprehensive and systematic verification and decision-making on the structured results, ensuring data credibility, result traceability, and the orderly operation of downstream application links.

[0053] Specifically, the implementation of data validation and business rule engines typically adopts a modular and strategic design approach. Taking mature product systems such as BPMN (Business Process Modeling and Representation), Drools (an open-source rule engine), or Alibaba Cloud Rule Engine as examples, the construction of a validation engine generally covers the following levels. First is basic format validation. Relying on the regular expressions, field type templates, length and value range controls embedded in the rule engine, each structured field in the invoice is validated for legality. For example, the invoice code field should be a highly formatted string with clearly defined length, character type, and value range; the date field must be formatted as "YYYY-MM-DD" and fall within the allowed range; and the amount field cannot contain negative values ​​or abnormal characters. This process is modeled as rule nodes, which can be flexibly maintained and extended, and are executed efficiently.

[0054] Secondly, business logic validation is more crucial in the validation process. This level of validation is achieved through a business rule base within the engine, maintained by domain business experts, covering major business scenarios and high-risk points. For example, the "amount in words" and "amount in figures" extracted from structured data must be absolutely consistent and conform to the rules for writing RMB; the "verification code" field must be able to automatically verify its legality based on fields such as "invoice code" and "invoice number" combined with a verification code generation algorithm; the "invoice date" must not be later than the "current date"; and the "total amount" of each item on the invoice must equal the sum of the individual item amounts to prevent tampering. For complex business scenarios, the rule engine can support advanced relationship validation between multiple fields, such as allowing only specific tax rates for certain types of special invoices, and automatically validating all business logic related to the license plate and engine number fields on motor vehicle invoices. Some validations even call external interfaces, such as connecting with the State Taxation Administration's invoice verification platform API to perform online verification of invoice uniqueness and voiding status to ensure that invoices are authentic, valid, and not duplicated.

[0055] The third implementation combines multiple validation modes, supporting dynamic policy configuration and visual business management. A mature validation and business rule engine includes rule priority, condition judgment, and dependency control. For example, rule-based inference engines like Drools support high concurrency and module independence in data validation flows through "if-then" rule sets, multi-level nesting, and condition chains. Business users can customize new rules, enable or disable certain rule nodes without changing the main program, enabling continuous system evolution and agile response. Meanwhile, penalty and fault tolerance mechanisms are also indispensable components of the system. When multiple business rule validations fail, the system automatically marks the associated structured data as "failed," pushes it to a manual review queue, and can perform statistical analysis on the types and frequency of abnormal data, providing feedback for front-end model optimization and business avoidance decisions.

[0056] In a specific implementation, during the construction of the Drools rule engine, all structured invoice information data is first input into the rule engine as Java objects or JSON containers. The system loads each business validation rule from the rule library. For example, if "the invoice code requires 12 digits," the rule is expressed as "when (invoice code.length==12) then (validation passed)." All fields are traversed and evaluated sequentially. For complex cross-field validations, the rule engine supports combined conditions and multi-path decisions, such as "the uppercase and lowercase amounts must be consistent; an anomaly is marked when the uppercase / lowercase matching algorithm fails." Furthermore, the Drools rule engine can also link fact objects with working memory to facilitate parallel and efficient handling of batch invoices, such as uniqueness checks within batch groups and cumulative merging logic.

[0057] Specifically, in step S7, if the verification result is successful, the structured invoice information data is output; if the verification result is unsuccessful, the structured invoice information data is sent to the manual intervention queue. Specifically, for the large number of frequently occurring standardized invoices, whose structural patterns, field formats, and business logic are highly standardized, the automation engine, after sufficient training, can accurately output structured data in most cases. Directly outputting the successful data achieves end-to-end automation, reduces manual intervention, and significantly improves business flow speed and operational efficiency. For example, downstream systems such as enterprise automated archiving, batch accounting in financial systems, online tax filing, and automatic accounting can seamlessly integrate automatic recognition results, efficiently realizing the digital flow of data and automated decision-making.

[0058] In actual business operations, even with optimized OCR and structured engines, automated systems cannot guarantee data integrity and standardization when faced with atypical invoices, scarce coding samples, abnormal data entry, format drift, damaged invoices, and ambiguous fields. Directly outputting data in such cases inevitably leads to data risks and may even cause significant errors in final accounting, tax reporting, and compliance audits. Therefore, when data verification fails, the structured information and related context of the invoice must be pushed to a dedicated human intervention queue. Experienced human auditors conduct a second review, correction, and supplementation to ensure that the data ultimately delivered to the main business system has sufficient accuracy and traceability, achieving efficient collaboration between machines and humans. The human intervention queue, on the one hand, blocks the automatic flow of suspicious or abnormal data, and on the other hand, continuously accumulates difficult scenarios and failed samples, providing valuable annotations and feedback for subsequent model retraining and rule base upgrades, accelerating the system's self-evolution capabilities.

[0059] In summary, the automatic identification, generation, and decision-making method for financial invoices provided in this application first receives and preprocesses the original invoice image. Then, it utilizes computer vision technology to perform layout analysis and classification on the enhanced image, distinguishing between known standard layouts and unknown layouts. For known layouts, a template-based OCR module is applied to accurately extract key text blocks; for unknown layouts, a general document understanding model is used to obtain preliminary key-value pairs of key information. Subsequently, the extracted information is transformed into structured data through a key information structure extraction module, and then verified by a data validation and business rule engine. If the verification passes, the structured invoice information is output; otherwise, it is sent to a manual intervention queue for further review. This method effectively improves the efficiency and accuracy of automated processing of financial invoices, while enhancing the system's flexibility in adapting to different invoice layouts.

[0060] This application also provides an automatic financial document recognition, generation, and decision-making system, such as... Figure 6As shown, the financial invoice automatic recognition, generation, and decision-making system 100 includes: an original invoice image receiving module 110, used to receive original invoice images uploaded by users; an original invoice image enhancement module 120, used to perform image preprocessing on the original invoice image to obtain an enhanced original invoice image; an original invoice format classification module 130, used to perform format analysis and classification on the enhanced original invoice image based on computer vision to obtain a format classification result; a region text block generation module 140, used to input the enhanced original invoice image into a template-based OCR module to obtain a region text block if the format classification result is a known standard format; and a preliminary key information key-value pair generation module 150, used to generate a key-value pair if the original invoice image is a known standard format. If the format classification result is unknown, the original enhanced image of the invoice is input into a general document understanding model to obtain preliminary key-value pairs. The invoice information extraction module 160 inputs the region text block or the preliminary key-value pairs into a key information structure extraction module to obtain structured invoice information data. The verification result generation module 170 inputs the structured invoice information data into a data verification and business rule engine to obtain a verification result. The invoice information output module 180 outputs the structured invoice information data if the verification result is successful. The manual intervention interface module 190 sends the structured invoice information data to a manual intervention queue if the verification result is unsuccessful.

[0061] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0062] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0063] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0064] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0065] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for automatic identification, generation, and decision-making of financial documents, characterized in that, include: Receive the original ticket image uploaded by the user; The original ticket image is preprocessed to obtain an enhanced original ticket image; The process of performing layout analysis and classification on the original enhanced invoice image based on computer vision to obtain layout classification results includes: extracting local and global visual features from the original enhanced invoice image using a deep convolutional neural network model to obtain local visual feature encoding vectors and global visual feature encoding vectors; combining the local and global visual feature encoding vectors to obtain a joint encoding vector of multi-scale visual features; and inputting the joint encoding vector of multi-scale visual features into an invoice type multi-classifier to obtain the layout classification results. Specifically, the process of combining the local visual feature encoding vector and the global visual feature encoding vector of the bill to obtain a joint encoding vector of multi-scale visual features of the bill includes: Information constraints are applied to each feature value of the local visual feature encoding vector and each feature value of the global visual feature encoding vector of the invoice to obtain the local visual feature encoding constraint vector and the global visual feature encoding constraint vector of the invoice, respectively, as follows: ;in, Represents the constraint vector for encoding local visual features of a bill. Each feature value, Represents the global visual feature encoding constraint vector of the invoice. Each characteristic value; The global information domain projection operator for the local visual feature encoding vector of the document and the global information domain projection operator for the global visual feature encoding vector of the document are calculated and expressed as follows: ;in, This represents the global information domain projection operator for the local visual feature encoding vector of the document. This represents the global information domain projection operator for the global visual feature encoding vector of the document; Based on the local visual feature encoding constraint vector of the bill, the global visual feature encoding constraint vector of the bill, the global information domain projection operator for the local visual feature encoding vector of the bill, and the global information domain projection operator for the global visual feature encoding vector of the bill, the local visual feature encoding vector and the global visual feature encoding vector of the bill are jointly regularized and fused to obtain the multi-scale visual feature joint encoding vector of the bill, expressed as: ;in, It is the feature combination function of the local visual feature encoding vector and the global visual feature encoding vector of the document, and It is the bias modulation vector. This indicates addition by position. This represents the joint encoding vector of the multi-scale visual features of the bill; If the format classification result is a known standard format, the original enhanced document image is input into a template-based OCR module to obtain regional text blocks; if the format classification result is an unknown format, the original enhanced document image is input into a general document understanding model to obtain preliminary key-value pairs. Input the region text block or the preliminary key information key-value pair into the key information structure extraction module to obtain structured ticket information data; The structured invoice information data is input into the data verification and business rule engine to obtain the verification result; If the verification result is successful, the structured invoice information data is output; if the verification result is unsuccessful, the structured invoice information data is sent to the manual intervention queue.

2. The method for automatic identification, generation, and decision-making of financial documents according to claim 1, characterized in that, Image preprocessing is performed on the original ticket image to obtain an enhanced original ticket image, including: The original ticket image is subjected to image denoising, tilt correction, brightness / contrast adjustment, and image binarization to obtain the enhanced image of the original ticket.

3. The method for automatic identification, generation, and decision-making of financial documents according to claim 1, characterized in that, If the format classification result is a known standard format, the original enhanced ticket image is input into a template-based OCR module to obtain region text blocks, including: Load a template file corresponding to the known standard layout, the template file including a layout identifier, a field list, and a region coordinate definition; Extract the region coordinate definition from the template file, and extract the image ROI region corresponding to each key information field from the original invoice enhancement image based on the region coordinate definition to obtain a set of key information field ROI images; Each key information field ROI image in the set of key information field ROI images is input into the OCR engine to obtain the region text block.

4. The method for automatic identification, generation, and decision-making of financial documents according to claim 3, characterized in that, Extracting region coordinate definitions from the template file, and extracting the image ROI region corresponding to each key information field from the original invoice enhancement image based on the region coordinate definitions, including: The original invoice enhancement image is image aligned so that the original invoice enhancement image is consistent with the coordinate system defined in the template file.

5. The method for automatic identification, generation, and decision-making of financial documents according to claim 1, characterized in that, If the format classification result is an unknown format, the original enhanced document image is input into a general document understanding model to obtain preliminary key-value pairs, including: The original ticket enhanced image is subjected to full-page OCR recognition to obtain all text and its location information; The original ticket enhancement image, along with all the text and its location information, is input into the trained general document understanding model to obtain the text sequence to be labeled; The text sequence to be labeled is converted into the preliminary key information key-value pairs.

6. A financial document automatic identification, generation, and decision-making system, used to execute the financial document automatic identification, generation, and decision-making method according to any one of claims 1-5, characterized in that, include: The original ticket image receiving module is used to receive original ticket images uploaded by users; The original invoice image enhancement module is used to perform image preprocessing on the original invoice image to obtain an enhanced original invoice image; The original invoice format classification module is used to perform format analysis and classification on the enhanced image of the original invoice based on computer vision to obtain format classification results. The region text block generation module is used to input the original ticket enhancement image into the template-based OCR module to obtain region text blocks if the format classification result is a known standard format. The preliminary key information key-value pair generation module is used to input the original ticket enhancement image into a general document understanding model to obtain preliminary key information key-value pairs if the format classification result is an unknown format. The invoice information extraction module is used to input the region text block or the preliminary key information key-value pair into the key information structure extraction module to obtain structured invoice information data; The verification result generation module is used to input the structured invoice information data into the data verification and business rule engine to obtain the verification result; The invoice information output module is used to output the structured invoice information data if the verification result is passed. The manual intervention interface module is used to send the structured ticket information data to the manual intervention queue if the verification result is unsuccessful.

Images