Invoice image recognition method and system based on deep learning

Abstract

This invention discloses a deep learning-based invoice image recognition method and system, belonging to the field of computer vision and artificial intelligence technology. The method includes the following steps: Intelligent routing decision step: Receiving the original invoice image and dynamically deciding whether to allocate it to a fast processing channel or a deep processing channel based on scene analysis results; Adaptive preprocessing step: Performing preprocessing operations of corresponding intensity on the original invoice image according to the allocation results of the intelligent routing decision step, obtaining a preprocessed image. This achieves a dynamic balance between efficiency and accuracy. Through intelligent routing decision, the system can dynamically select the processing path based on image complexity. Most simple invoices are processed quickly via the fast channel to meet high-concurrency scenarios; a few complex invoices are processed in the deep channel for detailed analysis to ensure high accuracy. This achieves optimal resource allocation at the system level.

Description

Technical Field

[0001] This invention relates to a method and system for recognizing invoice images based on deep learning, belonging to the field of computer vision and artificial intelligence technology. Background Technology

[0002] Invoices serve as core vouchers for financial reimbursement, tax accounting, and corporate auditing. The rapid, accurate, and automated recognition of invoice information is a crucial aspect of financial digitalization and intelligent transformation. Traditional invoice recognition methods primarily rely on optical character recognition (OCR) technology, typically employing a fixed processing pipeline: image preprocessing, layout analysis, character segmentation, OCR recognition, and post-processing verification.

[0003] The existing solution has the following shortcomings in practical use: Rigid workflows struggle to balance efficiency and accuracy, and single pipelines are ill-suited to varying image quality. Complex processing of high-quality images impacts speed, while simplistic processing of low-quality images reduces accuracy. The recognition results have low reliability, rely on a single OCR model, and lack multi-dimensional verification. The model cannot self-correct when it misidentifies, making it difficult to meet financial audit requirements. The system has poor adaptability and high maintenance costs. The new version of the invoice requires manual sample collection, labeling, and training, resulting in long iteration cycles and high costs. Lacking explainability and audit support, the solution operates in a black box, failing to provide a basis for decision-making and making it difficult to trace the audit in case of disputes. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for recognizing invoice images based on deep learning, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: Compared to existing technologies, the present invention provides a deep learning-based invoice image recognition method, characterized by the following steps: S1. Intelligent routing decision-making steps: Receive the original invoice image, and based on the scene analysis results of the original invoice image, dynamically decide to allocate it to the fast processing channel or the deep processing channel; S2. Adaptive preprocessing step: Based on the allocation result of the intelligent routing decision step, perform a preprocessing operation of corresponding intensity on the original invoice image to obtain the preprocessed image; S3. Multimodal unified recognition step: The preprocessed image is input into the corresponding recognition engine. The recognition engine performs end-to-end analysis on the invoice image based on a unified deep learning model and outputs preliminary structured data containing text content, location information and confidence level of at least one field. S4. In-depth trusted verification step: Perform multi-level verification on the preliminary structured data, including at least multi-source cross-validation and business logic verification, to generate final structured data and associated trusted evidence package; the trusted evidence package contains audit information for tracing the identification process; S5. Feedback evolution step: Collect low-confidence samples, verification conflict samples, or new version samples generated during the processing, and perform secure incremental learning on the recognition engine based on the samples to update the model parameters.

[0006] Furthermore, step S1 specifically includes: S101. Perform quality assessment and layout familiarity analysis on the original invoice image to obtain a quality score and a layout similarity score; S102. Based on the quality score, the layout similarity score, and the preset system load threshold, the original invoice image is allocated to the fast processing channel or the deep processing channel through intelligent routing hub decision-making. The fast processing channel is used to process regular invoices with high image quality and familiar layout, while the deep processing channel is used to process complex invoices with low image quality, unfamiliar layout, or high credibility requirements.

[0007] Furthermore, in step S2, the preprocessing operation performed on the original invoice image assigned to the depth processing channel includes at least one of the following: Adaptive illumination enhancement processing based on conditional generative adversarial networks; Geometric deformation correction processing based on differentiable spatial transformation network; Interference area removal and text restoration based on seal detection and repair network; Adversarial noise reduction based on frequency domain attention mechanism.

[0008] Furthermore, in step S3, the recognition engine includes a lightweight unified model and a complete unified model; The preprocessed image assigned to the fast processing channel is processed by the lightweight unified model; The preprocessed image assigned to the depth processing channel is processed by the complete unified model, and the character-level detection-segmentation-recognition integrated network is activated to generate character-level atomic data. The unified model is a multi-task model based on the Transformer architecture, which jointly completes invoice quality assessment, layout classification, semantic region detection, and text content generation in one forward propagation.

[0009] Furthermore, the multi-source cross-validation in step S4 includes: The text content of at least one key field in the preliminary structured data is compared with the corresponding field content parsed from the QR code area of ​​the original invoice image; Based on the comparison results of the confidence level of the key field in the preliminary structured data with the preset threshold, and the comparison results, a predetermined conflict resolution strategy is executed, which includes direct adoption, confidence enhancement, or marking conflict.

[0010] Furthermore, the business logic verification in step S4 includes: Input multiple numerical fields from the initial structured data into a pre-built financial knowledge graph or rule engine; Based on the business logic rules defined in the financial knowledge graph or rule engine, the relationship between the multiple numerical fields is validated, and the business logic rules include at least the validation of the reconciliation relationship between price and tax total.

[0011] Furthermore, the credible evidence package generated in step S4 also includes at least one of the following visual evidence: The heatmap showing the location of each field in the preliminary structured data on the original invoice image; The character-level segmentation mask output by the integrated character-level detection-segmentation-recognition network; The process log of multi-source cross-validation and business logic verification.

[0012] Furthermore, step S5 specifically includes: S501. New version samples are automatically discovered through a three-order trusted clustering verification mechanism. The three-order trusted clustering verification includes model prediction consistency verification, geometric rule matching verification, and cross-sample logical self-consistency verification. S502. Input the collected samples and the gold label samples that have been manually verified in step S4 into the security incremental learning sandbox. S503. In the secure incremental learning sandbox, the recognition engine model is incrementally trained using a noise-resistant collaborative teaching training framework to obtain an updated model. S504. Perform A / B testing on the updated model and the current production model, and deploy it to the production environment via hot update after passing the test.

[0013] A deep learning-based invoice image recognition system includes: Furthermore, the intelligent access and scheduling module is used to receive the original invoice image and dynamically route it based on the scene analysis results; An adaptive preprocessing module is used to perform differential preprocessing on the image based on the routing results; The intelligent sensing and processing module includes a lightweight recognition engine corresponding to the fast processing channel and a complete recognition engine corresponding to the deep processing channel, which is used to output preliminary structured data; The business verification and trust assurance module is used to perform multi-level verification on the preliminary structured data and generate the final structured data and trust evidence package. The co-evolution and feedback closed-loop module is used to collect samples and drive the secure incremental updates of the recognition engine.

[0014] Furthermore, the intelligent access and scheduling module includes an intelligent routing hub, which integrates a lightweight scene classifier to output scene labels to drive routing decisions; The system supports the collaborative deployment of central cloud, edge computing nodes and end devices. The lightweight recognition engine is deployed on edge computing nodes or end devices, while the complete recognition engine and the business verification and trust assurance module are deployed on the central cloud.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: A dynamic balance between efficiency and accuracy is achieved. Through intelligent routing decisions, the system can dynamically select the processing path based on image complexity. Most simple invoices are processed at high speed via the fast channel to meet high-concurrency scenarios; a few complex invoices are processed in the deep channel for detailed analysis to ensure high accuracy. Optimal resource allocation is achieved at the system level.

[0016] A multi-layered trusted output system was constructed, forming a defense-in-depth verification mechanism by introducing multi-source cross-validation (OCR and QR codes), business logic verification, and visual evidence generation. This not only significantly improves the accuracy of the recognition results but also makes the output results interpretable and auditable, meeting financial compliance requirements.

[0017] It endows the system with the ability to continuously self-evolve: through a three-order trusted clustering mechanism, new versions are automatically discovered, and combined with a secure incremental learning closed loop, the system can continuously learn from actual business data, automatically adapt to changes, shorten the model iteration cycle from days to hours, and significantly reduce operation and maintenance costs and response latency.

[0018] It provides a complete engineering solution: This invention not only proposes an innovative method, but also defines the corresponding system architecture, supports cloud, edge, and terminal collaborative deployment, has good scalability and practicality, and can be directly integrated into the enterprise's existing financial processes to generate immediate business value. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a system architecture diagram of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Please see Figure 1-2 The present invention provides a technical solution: A deep learning-based invoice image recognition method includes, S1, Intelligent Routing Decision Steps This step corresponds to step S1 in claim 1 and is the starting point for the system to implement dynamic resource allocation and differentiated processing. Its purpose is to quickly "diagnose" the input image and assign it an appropriate processing path while consuming minimal computing resources.

[0023] S101, Multi-source image access and metadata tagging. The system receives original invoice images from different channels through a unified API gateway. Simultaneously, it receives or generates business metadata related to the image. The system generates a globally unique tracking identifier for this processing task. .

[0024] S102, Lightweight and Fast Scene Feature Extraction. The system uses a lightweight convolutional neural network as the scene classifier. .

[0025] Input: Original image Scaling to a fixed low resolution, denoted as .

[0026] deal with: pass Output a multidimensional feature vector .

[0027] Output: A parallel fully connected branch outputs the initial quality score. The calculation formula is as follows: in, yes Global average pooling characteristics of the intermediate layer and For learnable parameters, This is the Sigmoid activation function.

[0028] S103, Scenario Analysis and Routing Decision. The process of making a decision is the core of intelligent routing, as it integrates multiple factors to make the final decision.

[0029] S1031. Layout Familiarity Calculation: Calculate Feature Vector Each baseline feature vector in the pre-stored "known version feature library" cosine similarity : The highest similarity value is taken as the layout familiarity score for the image. If the highest similarity is lower than the preset unfamiliarity threshold... If so, it is judged to be a suspected new version.

[0030] S1032, Generation of Comprehensive Decision Factors: Decision Factors It is a comprehensive scalar used to quantify the expected "complexity" required to process the image. An example calculation formula is: in: The lower the quality, the larger the factor value.

[0031] The more unfamiliar the layout, the larger the factor value.

[0032] Business priority weight.

[0033] : Configurable weighting coefficients, satisfying .

[0034] S1033, Dynamic Routing Decision: The intelligent routing hub maintains a dynamic system load status. The final channel allocation decision rules are as follows: Forced depth channel: If satisfied If the complexity threshold is high or the business requires high reliability, it will be forcibly assigned to the deep processing channel.

[0035] Fast track to load awareness: If (Low complexity threshold) and system load If the load threshold is low, it will be assigned to the fast processing channel.

[0036] Default or queued decision: In other cases, allocation is based on preset strategies or real-time load.

[0037] S1034. Routing Information Encapsulation: Generating Routing Decision Results .

[0038] S2, Adaptive Preprocessing Steps This step corresponds to step S2 in claim 1, based on the routing decision result. For the original image Perform preprocessing operations with different strengths and strategies.

[0039] S201, Basic Geometric Correction and Normalization.

[0040] S2011-S2013, Fusion Correction: via QR code angle and table line angle Calculate the final correction angle : Among them, weight It depends on the confidence level of the QR code detection. Apply rotation correction, with a rotation angle of 100°. .

[0041] S2014, Size Normalization: Obtain the basic corrected image .

[0042] S202, Deep Enhancement Processing.

[0043] For images assigned to the depth channel, Based on this, a series of enhancement sub-modules are initiated. This sub-step corresponds to the content of claim 3.

[0044] S2021, Adaptive Illumination Enhancement: Generator by Input, output brightness adjustment diagram Adjusted image .

[0045] S2022, Nonlinear Deformation Correction: A network predicts the dense flow field, and image rearrangement is achieved through bilinear sampling to obtain a corrected image. .

[0046] S2023, Seal and Interference Handling: Using a Segmentation Model Generate mask Repair the network Output the repaired image .

[0047] S2024, Adversarial Sanitization: Noise is suppressed in the frequency domain using an attention module to obtain a sanitized image. .

[0048] S2025, Quality Verification: Calculate Depth Preprocessed Image Quality rating .

[0049] S203, Encapsulation of preprocessing results.

[0050] For the fast channel, the preprocessed image is .

[0051] For the depth channel, the preprocessed image is .

[0052] S303, Preliminary generation of structured data.

[0053] Regardless of the speed of the channel, this step ultimately generates a unified preliminary structured data object. .

[0054] S4, Steps for in-depth trusted verification This step corresponds to step S4 in claim 1 and is the core of ensuring that the output results are credible, reliable, and in line with business logic.

[0055] S401, Multi-source cross-validation.

[0056] This sub-step corresponds to the content of claim 5.

[0057] S4011, QR code information parsing: obtaining structured data .

[0058] S4012, Field-level comparison and conflict resolution: Execute the rule engine. For example, for confidence levels... Fields: like If so, the OCR result will be adopted. .

[0059] like And the QR code value exist: If they match, the OCR result will be adopted, and its credibility will be increased. .

[0060] If they are inconsistent, mark them as conflicting.

[0061] like and If it exists, it will be adopted first. , .

[0062] S402, Business Logic Validation. This sub-step corresponds to the content of claim 6.

[0063] S4021, Knowledge Graph Query: Link key entity fields to the enterprise financial knowledge graph.

[0064] S4022, Numerical Logic Verification: Verification of Amount ,tax rate ,tax Total price including tax Perform verification: in This is for the allowable minute error.

[0065] S4023, Business Rule Engine: Executes configurable business rules.

[0066] S403, Generate a trusted evidence package. This sub-step corresponds to the content of claim 7.

[0067] S4031, Visual Evidence Generation: Generating Field Location Heatmaps And character-level masks.

[0068] S4032. Structured Audit Log Generation: Generate detailed audit logs. .

[0069] S4033, Digital Signature and Encapsulation: Calculating Hash Values And generate a signature Encapsulated as a trusted evidence package .

[0070] S404, Final Result Synthesis and Output.

[0071] Based on the comprehensive verification results, the final structured data is generated. .

[0072] S5, Feedback Evolution Steps This step corresponds to step S5 in claim 1, enabling the system to continuously learn from operational data and optimize itself.

[0073] S501, Automatic collection of evolutionary samples. Collect low-confidence data. , , verification of conflicts and manual review of samples.

[0074] S502, Third-order Trustworthy Clustering and New Knowledge Discovery. This sub-step corresponds to the content regarding third-order reliable clustering in claim 8.

[0075] S5021, Clustering: Clustering of sample feature vectors Perform cluster analysis.

[0076] S5022, Third-order verification: For clusters Verification required: Model consistency verification: The highest probability that a sample within a cluster is predicted by the model to be of a known pattern. .

[0077] Geometric rule verification: Samples within a cluster conform to the general spatial constraints.

[0078] Cross-sample logical self-consistency verification: There are no logical contradictions between samples within the same cluster.

[0079] S5023, New Knowledge Confirmation: Clusters that pass verification are marked as "Trusted New Clusters".

[0080] S503, Incremental learning for security. This sub-step corresponds to the content regarding secure incremental learning in claim 8.

[0081] S5031. Sample preparation and cleaning: Data is cleaned using a collaborative teaching framework.

[0082] S5032, Isolation Training: Incremental training is performed in a training sandbox, and algorithms such as elastic weight solidification are used to mitigate catastrophic forgetting.

[0083] S5033, Evaluation and Release: New Model Evaluation on the test set showed that only key metrics decreased by no more than [a certain percentage]. The release process only begins when the percentage reaches 0.5% (e.g., 0.5%).

[0084] S504, Knowledge Base Update. The "Known Format Feature Library", financial knowledge graph and business rule engine are updated simultaneously.

[0085] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A deep learning-based invoice image recognition method, characterized in that, Includes the following steps: S1. Intelligent routing decision-making steps: Receive the original invoice image, and based on the scene analysis results of the original invoice image, dynamically decide to allocate it to the fast processing channel or the deep processing channel; S2. Adaptive preprocessing step: Based on the allocation result of the intelligent routing decision step, perform a preprocessing operation of corresponding intensity on the original invoice image to obtain the preprocessed image; S3. Multimodal unified recognition step: The preprocessed image is input into the corresponding recognition engine. The recognition engine performs end-to-end analysis on the invoice image based on a unified deep learning model and outputs preliminary structured data containing text content, location information and confidence level of at least one field. S4. In-depth trusted verification step: Perform multi-level verification on the preliminary structured data, including at least multi-source cross-validation and business logic verification, to generate the final structured data and the associated trusted evidence package; the trusted evidence package contains audit information for tracing the identification process; S5. Feedback evolution step: Collect low-confidence samples, verification conflict samples, or new version samples generated during the processing, and perform secure incremental learning on the recognition engine based on the samples to update the model parameters.

2. The invoice image recognition method based on deep learning according to claim 1, characterized in that, Step S1 specifically includes: S101. Perform quality assessment and layout familiarity analysis on the original invoice image to obtain a quality score and a layout similarity score; S102. Based on the quality score, the layout similarity score, and the preset system load threshold, the original invoice image is allocated to the fast processing channel or the deep processing channel through intelligent routing hub decision-making. The fast processing channel is used to process regular invoices with high image quality and familiar layout, while the deep processing channel is used to process complex invoices with low image quality, unfamiliar layout, or high credibility requirements.

3. The invoice image recognition method based on deep learning according to claim 1, characterized in that, In step S2, the preprocessing operation performed on the original invoice image assigned to the depth processing channel includes at least one of the following: Adaptive illumination enhancement processing based on conditional generative adversarial networks; Geometric deformation correction processing based on differentiable spatial transformation network; Interference area removal and text restoration based on seal detection and repair network; Adversarial noise reduction based on frequency domain attention mechanism.

4. The invoice image recognition method based on deep learning according to claim 1, characterized in that, In step S3, the recognition engine includes a lightweight unified model and a complete unified model; The preprocessed image assigned to the fast processing channel is processed by the lightweight unified model; The preprocessed image assigned to the depth processing channel is processed by the complete unified model, and the character-level detection-segmentation-recognition integrated network is activated to generate character-level atomic data. The unified model is a multi-task model based on the Transformer architecture, which jointly completes invoice quality assessment, layout classification, semantic region detection, and text content generation in one forward propagation.

5. The invoice image recognition method based on deep learning according to claim 1, characterized in that, The multi-source cross-validation in step S4 includes: The text content of at least one key field in the preliminary structured data is compared with the corresponding field content parsed from the QR code area of ​​the original invoice image; Based on the comparison results of the confidence level of the key field in the preliminary structured data with the preset threshold, and the comparison results, a predetermined conflict resolution strategy is executed, which includes direct adoption, confidence enhancement, or marking conflict.

6. The invoice image recognition method based on deep learning according to claim 1, characterized in that, The business logic verification in step S4 includes: Input multiple numerical fields from the initial structured data into a pre-built financial knowledge graph or rule engine; Based on the business logic rules defined in the financial knowledge graph or rule engine, the relationship between the multiple numerical fields is validated, and the business logic rules include at least the validation of the reconciliation relationship between price and tax total.

7. The invoice image recognition method based on deep learning according to claim 1, characterized in that, The credible evidence package generated in step S4 also includes at least one of the following visual evidence: The heatmap showing the location of each field in the preliminary structured data on the original invoice image; The character-level segmentation mask output by the integrated character-level detection-segmentation-recognition network; The process log of multi-source cross-validation and business logic verification.

8. The invoice image recognition method based on deep learning according to claim 1, characterized in that, Step S5 specifically includes: S501. New version samples are automatically discovered through a three-order trusted clustering verification mechanism. The three-order trusted clustering verification includes model prediction consistency verification, geometric rule matching verification, and cross-sample logical self-consistency verification. S502. Input the collected samples and the gold label samples that have been manually verified in step S4 into the security incremental learning sandbox. S503. In the secure incremental learning sandbox, the recognition engine model is incrementally trained using a noise-resistant collaborative teaching training framework to obtain an updated model. S504. Perform A / B testing on the updated model and the current production model, and deploy it to the production environment via hot update after passing the test.

9. The invoice image recognition system based on deep learning according to claim 1, characterized in that, include: The intelligent access and scheduling module is used to receive the original invoice images and dynamically route them based on the scene analysis results; An adaptive preprocessing module is used to perform differential preprocessing on the image based on the routing results; The intelligent sensing and processing module includes a lightweight recognition engine corresponding to the fast processing channel and a complete recognition engine corresponding to the deep processing channel, which is used to output preliminary structured data; The business verification and trust assurance module is used to perform multi-level verification on the preliminary structured data and generate the final structured data and trust evidence package. The co-evolution and feedback closed-loop module is used to collect samples and drive the secure incremental updates of the recognition engine.

10. The invoice image recognition system based on deep learning according to claim 1, characterized in that, The intelligent access and scheduling module includes an intelligent routing hub, which integrates a lightweight scene classifier to output scene labels to drive routing decisions. The system supports the collaborative deployment of central cloud, edge computing nodes and end devices. The lightweight recognition engine is deployed on edge computing nodes or end devices, while the complete recognition engine and the business verification and trust assurance module are deployed on the central cloud.

Images