A financial report image data extraction and verification method, system and storage medium

Abstract

The application provides a kind of financial report image data extraction and verification method, system and storage medium;The method detects and desensitizes sensitive information area of financial report image locally, calls cloud OCR to obtain recognition result after performing seal removal processing;Based on report type, load accounting formula rule library to calculate residual error, and calculate comprehensive score combined with OCR confidence and image quality;When the comprehensive score is lower than the threshold, trigger local rollback and re-identify, slice the original image in high definition for secondary identification and result fusion according to residual error minimization;At the same time, standardized mapping is performed on accounting subjects, and structured data is output by completing verification combined with integrity and cross-checking relationship.The application realizes a closed-loop mechanism of "local desensitization + cloud OCR + accounting formula residual error verification + threshold triggering local rollback and re-identification + subject standardization", which improves the accuracy, reliability and processing efficiency of financial report image structured extraction under the premise of ensuring that sensitive data does not leave the local.

Description

Technical Field

[0001] This invention relates to the field of financial data information technology, specifically to a method, system, and storage medium for extracting and verifying image data from financial reports. Background Technology

[0002] In business scenarios such as financial credit approval, corporate risk control, investment analysis, and regulatory technology, financial reports (hereinafter referred to as "reports") are important bases for measuring a company's financial health and credit risk. Reports typically include key financial statements such as balance sheets, income statements, and cash flow statements, and their authenticity, completeness, and availability directly affect credit decisions, risk pricing, and compliance reviews.

[0003] However, in actual business processes, reports are often stored and circulated as scanned PDFs or images, making the process of "converting unstructured report images into structured data" a bottleneck in the industry. These report images also commonly suffer from issues such as obscured seals / signatures, inconsistent layouts, and diverse subject names, further increasing the difficulty of automatic recognition and structuring.

[0004] The shortcomings of existing technology: 1) High data security and compliance risks make it difficult to meet regulatory requirements for financial scenarios. Financial reports contain highly sensitive information such as core financial data, business secrets, legal representative information, and signatures. Current technologies that rely on public cloud OCR APIs to process images inevitably lead to the leakage of sensitive data, including company seals and legal representative signatures, to third-party service providers, resulting in significant compliance and data breach risks.

[0005] 2) Non-standard report formats and layouts, template / fixed area OCR is prone to failure. Financial reports issued by different companies and institutions have varying formats, column widths, and header designs; "the format varies from company to company." This makes it difficult for traditional OCR methods that rely on template matching or fixed area positioning to accurately locate indicator names and corresponding values, resulting in poor stability of structured extraction.

[0006] 3) The lack of a unified standard for subject naming leads to semantic mapping failure. The same financial indicator may be expressed in multiple ways in different reports (e.g., multiple expressions of "net profit"). Existing OCR usually only solves "character recognition" and has difficulty understanding "semantic consistency". As a result, the recognized account text cannot be reliably mapped to the standard account / standard report template inside the system, resulting in low structure quality and usability.

[0007] 4) Obscuring of seals / signatures and image contamination can lead to the loss or misidentification of critical information. Financial reports often require a seal or signature, and obscuring objects frequently cover key account names or financial figures. Current technologies for handling obscuring often employ a method of "identifying the obscured area and discarding the obscured pixel information," relying solely on residual strokes for recognition. This is particularly problematic for closely spaced numbers, easily leading to data extraction errors.

[0008] In addition, poor scanning quality and blurry fonts also cause OCR to have an inherent misrecognition rate of numbers (such as confusion between "8 / 6" and "1 / 7"), further amplifying the risk of incorrect input.

[0009] 5) The lack of an automated verification loop based on accounting standards makes it difficult to detect and correct errors in a timely manner. Existing technologies often lack automated, explainable, and traceable accounting logic verification mechanisms; once erroneous data is identified and entered into the system, it will cause "garbage in, garbage out," which will have a continuous negative impact on subsequent risk control and decision-making.

[0010] 6) It is difficult to achieve a balance between "safety, efficiency, and accuracy," resulting in structural contradictions in practical applications. In summary, existing technologies for financial report image processing are prone to falling into the "impossible triangle": pursuing efficiency often relies on cloud capabilities at the expense of security; pursuing security tends to be purely local processing at the expense of processing efficiency and recognition accuracy; and multiple automation solutions still cannot simultaneously solve the persistent accuracy problems caused by seal obscuring and non-standard account categories.

[0011] Therefore, existing technologies have shortcomings and need further improvement. Summary of the Invention

[0012] To address the problems existing in the prior art, the present invention provides a method, system, and storage medium for extracting and verifying image data from financial reports.

[0013] To achieve the above objectives, the specific solution of the present invention is as follows: This invention provides a method for extracting and verifying image data from financial reports, comprising: S1. Obtain the financial report image to be processed, and perform sensitive information region detection on the financial report image locally to obtain a pixel-level mask of the sensitive information region. The pixel-level mask includes at least a seal region mask and a signature region mask. S2. Perform desensitization and seal removal processing on the financial report image based on the pixel-level mask: perform desensitization processing on the financial report image based on the signature area mask to generate a desensitized financial report image; and perform seal removal processing on the desensitized financial report image based on the seal area mask to generate a seal-removed financial report image; wherein, the original undesensitized financial report image is only retained locally and is not transmitted over the network; S3. Send the image of the financial report without seal to the cloud OCR server for recognition and obtain the cloud recognition result. The cloud recognition result includes at least the recognized text of the table cell, cell coordinate information and the corresponding OCR confidence score. S4. Determine the report type based on the cloud recognition result and load the accounting formula rule set corresponding to the report type. Perform accounting formula calculation on the values ​​in the cloud recognition result to obtain at least one accounting formula residual. S5. For at least one numerical cell, calculate the comprehensive score of the numerical cell based on the OCR confidence of the numerical cell, the image quality score of the numerical cell, and the residual penalty term corresponding to the accounting formula residual, and compare the comprehensive score with a preset score threshold. S6. When the overall score is lower than the preset score threshold, trigger local rollback and re-identification: crop a high-definition slice corresponding to the target cell from the original un-anonymized financial report image, call the local OCR engine to perform secondary recognition on the high-definition slice to obtain the local recognition result, and select or fuse the results of the local recognition result and the cloud recognition result based on the residual minimization criterion, and output the final value of the target cell; when the overall score is not lower than the preset score threshold, use the cloud recognition result as the final value of the target cell; S7. Perform accounting subject normalization on the identified accounting subject text: map the accounting subject text to the standard accounting subjects in the standard accounting subject library; and after completing the accounting subject normalization, perform integrity verification and accounting formula calculation verification on the structured financial data, and output the structured financial data and verification results.

[0014] Furthermore, the sensitive information region detection uses a pre-trained target detection model to scan and locate the financial report image and generate the pixel-level mask. The pixel-level mask includes at least a seal region mask and a signature region mask. The desensitization process includes performing Gaussian blur processing and / or zero-filling processing on the image region corresponding to the signature region mask.

[0015] Furthermore, the stamp removal process includes performing image inpainting and reconstruction on the stamp-occluded area to restore the obscured character or number stroke information; the image inpainting and reconstruction is implemented using a generative adversarial network model and / or a content-aware image inpainting model.

[0016] Furthermore, the set of accounting formula rules includes at least intra-statement balance verification rules and cross-statement reconciliation verification rules; wherein, the intra-statement balance verification rules include at least verification of the identity relationship between total assets, total liabilities, and total owners' equity, and the cross-statement reconciliation verification rules include at least verification of the reconciliation relationship between net profit in the income statement and retained earnings in the balance sheet. The image quality score is obtained by performing a sharpness calculation on the image region corresponding to the target cell. The sharpness calculation includes a sharpness assessment based on the Laplacian operator or the gradient operator. The comprehensive score is a weighted combination of OCR confidence, image quality score, and residual penalty term. The residual penalty term is penalized when the residual of the accounting formula involving the target cell is greater than zero, and is not penalized when the residual of the accounting formula is zero. The result selection or fusion includes: comparing the cloud recognition result with the local recognition result, and preferentially selecting the value that reduces the residual of at least one accounting formula involving the target cell the most as the final value; when multiple candidate cells jointly affect the same accounting formula, the final values ​​of multiple candidate cells are jointly selected or iteratively updated according to the residual minimization criterion.

[0017] This invention also provides a financial report image data extraction and verification system, deployed on one or more computing devices, for implementing the above method, including: The document interface module is used to receive financial report image inputs and output structured financial data and a manual review interface; The preprocessing module is used to perform sensitive information area detection locally to obtain the seal area mask and signature area mask, and to perform desensitization processing on the financial report image based on the signature area mask. At the same time, it performs seal removal processing on the desensitized financial report image based on the seal area mask to generate a seal-removed financial report image. The hybrid OCR module includes a cloud-based OCR interface and a local OCR engine. The cloud-based OCR interface is used to recognize the image of the financial report without the seal and output the cloud recognition result, which includes at least the cell recognition text, cell coordinate information and OCR confidence. The local OCR engine is used to backtrack and re-recognize the high-definition slice of the original un-de-sensitized image when the comprehensive score is lower than the threshold. The accounting verification module includes an accounting formula rule library, which is used to execute accounting formula calculations and output accounting formula residuals, and to calculate a comprehensive score to trigger local rollback and re-identification; The accounting subject standardization module is used to map the identified accounting subject text to a standard accounting subject library and generate structured financial data; The data storage module is used to store the standard accounting subject library, accounting formula rule library, mapping rules, and structured financial data and verification results.

[0018] Furthermore, the accounting subject standardization module includes a synonym extension set construction unit and a dual similarity calculation unit; the dual similarity calculation unit is used to calculate the literal similarity and semantic similarity between the accounting subject text and the standard accounting subject, and weight the two to obtain a mapping score; when the mapping score is lower than the subject mapping threshold, the accounting subject text is marked as an item to be manually confirmed and pushed to the manual review interface.

[0019] Furthermore, it also includes an accounting expert intelligent agent module, which is a rule-based expert system; when the accounting verification module reports that the accounting formula calculation has failed, the accounting expert intelligent agent module traces back the accounting subject data related to the failed formula based on the accounting rule knowledge base, locates one or more numerical cells suspected of being identified incorrectly, and generates repair suggestions for manual review.

[0020] Furthermore, it also includes a Trusted Execution Environment (TEE) module and a formal constraint solver module; The Trusted Execution Environment (TEE) module is used to perform sensitive information region detection, desensitization, and seal removal processing of the preprocessing module within the Trusted Execution Environment; high-definition slice cropping of the original undesensitized image required for local rollback and re-identification of the hybrid OCR module; and local OCR inference. Before sending the seal-removed financial report image to the cloud, it generates a remote verification report bound to the metric value and program version of the Trusted Execution Environment. The cloud OCR interface in the hybrid OCR module only receives the seal-removed financial report image and returns the cloud recognition result when the remote verification report is verified successfully. The formal constraint solver module is used to formalize the accounting formula rule base in the accounting verification module into a constraint set, and construct a candidate value set for discrete variables by combining the cloud recognition results and the candidate values ​​output by the local OCR engine. An optimization model is established with the objective function of minimizing the accounting formula residual and minimizing the number of replaced cells and / or the replacement cost. The optimal solution that satisfies the constraint set is obtained by solving the model using Satisfiability Modulo Theories (SMT) or integer linear programming, so as to determine the final value of each target cell for outputting structured financial data.

[0021] Furthermore, it also includes a digital forensics anti-forgery detection module, which is configured to perform tampering / forgery risk analysis on the financial report image before or after the preprocessing module outputs the image of the financial report with the seal removed, including at least one or more of the following detections: compression consistency detection based on error level analysis (ELA), compression parameter consistency detection based on JPEG quantization table / block effect, consistency detection based on image noise fingerprint (PRNU), and splicing anomaly detection based on layout geometric consistency and font / character spacing statistical features, and output an evidence risk score R_forensic; Specifically, when calculating the comprehensive score, the accounting verification module incorporates the evidence collection risk score R_forensic as a risk penalty item, so that the comprehensive score simultaneously reflects OCR confidence, image quality score, accounting formula residuals, and evidence collection risk. Furthermore, when the evidence collection risk score R_forensic reaches or exceeds a preset evidence collection threshold, at least one handling strategy is triggered: performing local rollback re-identification on the target cell related to the accounting formula residual, marking the corresponding structured financial data as pending manual review, and / or outputting tampering risk warning information, thereby further suppressing the risk of erroneous data entry caused by forged or tampered images on the basis of closed-loop verification of accounting residuals.

[0022] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a method for extracting and verifying financial report image data.

[0023] The technical solution of this invention has the following beneficial effects: 1. Enhanced data security, meeting financial compliance requirements that "highly sensitive identity information such as signatures and seals must not leave the local area." This invention first performs sensitive information area detection locally and generates pixel-level masks to mask and desensitize sensitive information such as seal areas and signature areas. The desensitized report image is then sent to the cloud for OCR recognition. Simultaneously, it explicitly states that "the original un-desensitized image is only stored locally and not transmitted over the network." Therefore, while utilizing cloud computing power to improve recognition efficiency, it avoids the leakage of original sensitive financial data and reduces data leakage and compliance risks introduced by third-party cloud services.

[0024] 2. It is more robust to the loss of key information caused by seals / occlusions, improving the completeness and accuracy of recognition. To address the common issue of "stamps obscuring numbers / text" in financial reports, this invention does not simply discard the obscuring pixels. Instead, it further performs stamp removal / image restoration processing (such as GAN-based restoration) on the desensitized image to restore the obscured strokes and character structures. This reduces the probability of missed or misidentified recognition caused by obscuring from the source, thereby improving the quality and integrity of subsequent OCR input.

[0025] 3. Introduce "accounting logic residuals" to achieve interpretable automatic verification and improve the credibility of structured data. After obtaining the structured OCR results from the cloud, this invention loads a set of accounting identities / accounting formula rules based on the identified report type and calculates the formula residual (Δ). Δ=0 indicates logical consistency, while Δ>0 indicates logical inconsistency. Since financial statements are inherently constrained by accounting identities and reconciliation relationships, introducing residuals as a verification signal can automatically detect and locate seemingly reasonable but actually erroneous OCR values, thereby improving the credibility of structured financial data.

[0026] 4. Multi-dimensional comprehensive scoring and threshold-triggered local backtracking and re-identification form an "automatic error correction closed loop," further improving the final accuracy. This invention constructs a comprehensive scoring model for each numerical cell, incorporating OCR confidence, image quality factors, and formula residuals into a unified evaluation. When the comprehensive score falls below a threshold, a key innovation is triggered—local backtracking and re-identification: high-resolution slices are cropped from the original, un-anonymized image at cell coordinates, and local OCR is used for secondary recognition. Values ​​that reduce or eliminate the formula residual Δ are prioritized as the final result. This achieves a closed-loop mechanism of "rapid initial recognition in the cloud + high-precision local fallback error correction + accounting logic constraint selection," reducing the systemic risk caused by numerical misidentification.

[0027] 5. AI-based subject standardization frees the system from rigid template dependence and enhances its adaptability to non-standard financial statements. To address the issue of inconsistent subject naming across different enterprises and institutions, this invention pre-constructs a standard accounting subject library and maintains an extended set of synonyms / near-synonyms. Simultaneously, it employs a dual similarity calculation and weighted decision-making process using both literal similarity (edit distance) and semantic similarity (vector cosine similarity) to robustly map the original subject text extracted by OCR to the standard subjects. This solution can simultaneously correct misidentification of similar-looking characters (literal level) and identify subjects with different expressions but the same meaning (semantic level), thereby improving the quality of subject normalization and enhancing its versatility across templates and formats.

[0028] 6. The accounting expert AI provides rule-driven remediation suggestions, reducing the cost of manual review and improving interpretability. When the formula calculation engine reports a verification failure, this invention activates the "accounting expert intelligent agent," which automatically backtracks the relevant subject data of the failed formula based on a preset accounting rule knowledge base, locates one or more values ​​that may have OCR recognition errors, and generates repair suggestions for manual verification. Because this intelligent agent is rule-based, it can output the basis for "why a certain cell is suspected and how to fix it," transforming manual review from a "full table check" to a "targeted check," reducing review costs and improving the interpretability of the processing.

[0029] 7. Integrity verification + intra-table balancing / cross-table reconciliation accounting reduces the risk of "missed retrieval / incorrect retrieval / inconsistent entry". After standardizing accounts and correcting errors, this invention performs data integrity verification and accounting formula verification. Accounting verification includes at least intra-statement balance verification (e.g., Assets = Liabilities + Owner's Equity) and cross-statement reconciliation verification (e.g., the relationship between net profit in the income statement and retained earnings in the balance sheet). Therefore, it can intercept missing items, abnormal items, and inconsistencies before data entry, improving the accuracy and consistency of the final output structured financial data.

[0030] 8. End-to-end automation improves overall governance efficiency and processing throughput, and has the value for large-scale implementation. This invention integrates desensitization, seal processing, OCR recognition, subject mapping, formula verification, and error correction into an end-to-end automated process, reducing manual intervention and shortening the processing chain. Compared with traditional manual processing, it improves governance efficiency and further enhances the automation rate, demonstrating good engineering application and commercial value. Attached Figure Description

[0031] Figure 1 This is a flowchart of the overall method of the present invention; Figure 2 This is a system architecture diagram of the present invention; Figure 3 This is a detailed diagram of the verification and local rollback re-identification mechanism of the present invention. Detailed Implementation

[0032] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings, and not all of them.

[0033] Combination Figures 1-3 As shown, the present invention provides a method for extracting and verifying image data from financial reports, comprising: S1. Obtain the financial report image to be processed, and perform sensitive information region detection on the financial report image locally to obtain a pixel-level mask of the sensitive information region. The pixel-level mask includes at least a seal region mask and a signature region mask. S2. Perform desensitization and seal removal processing on the financial report image based on the pixel-level mask: perform desensitization processing on the financial report image based on the signature area mask to generate a desensitized financial report image; and perform seal removal processing on the desensitized financial report image based on the seal area mask to generate a seal-removed financial report image; wherein, the original undesensitized financial report image is only retained locally and is not transmitted over the network; S3. Send the image of the financial report without seal to the cloud OCR server for recognition and obtain the cloud recognition result. The cloud recognition result includes at least the recognized text of the table cell, cell coordinate information and the corresponding OCR confidence score. S4. Determine the report type based on the cloud recognition result and load the accounting formula rule set corresponding to the report type. Perform accounting formula calculation on the values ​​in the cloud recognition result to obtain at least one accounting formula residual. S5. For at least one numerical cell, calculate the comprehensive score of the numerical cell based on the OCR confidence of the numerical cell, the image quality score of the numerical cell, and the residual penalty term corresponding to the accounting formula residual, and compare the comprehensive score with a preset score threshold. S6. When the overall score is lower than the preset score threshold, trigger local rollback and re-identification: crop a high-definition slice corresponding to the target cell from the original un-anonymized financial report image, call the local OCR engine to perform secondary recognition on the high-definition slice to obtain the local recognition result, and select or fuse the results of the local recognition result and the cloud recognition result based on the residual minimization criterion, and output the final value of the target cell; when the overall score is not lower than the preset score threshold, use the cloud recognition result as the final value of the target cell; S7. Perform accounting subject normalization on the identified accounting subject text: map the accounting subject text to the standard accounting subjects in the standard accounting subject library; and after completing the accounting subject normalization, perform integrity verification and accounting formula calculation verification on the structured financial data, and output the structured financial data and verification results.

[0034] The sensitive information region detection uses a pre-trained target detection model to scan and locate the financial report image and generate the pixel-level mask. The pixel-level mask includes at least a seal region mask and a signature region mask. The desensitization process includes performing Gaussian blur processing and / or zero-filling processing on the image region corresponding to the signature region mask.

[0035] The stamp removal process includes performing image inpainting and reconstruction on the stamp-occluded area to restore the obscured character or number stroke information; the image inpainting and reconstruction is implemented using a generative adversarial network model and / or a content-aware image inpainting model.

[0036] The accounting formula rule set includes at least intra-statement balance verification rules and cross-statement reconciliation relationship verification rules; wherein, the intra-statement balance verification rules include at least the verification of the identity relationship between total assets, total liabilities and total owners' equity, and the cross-statement reconciliation relationship verification rules include at least the verification of the reconciliation relationship between net profit in the income statement and retained earnings in the balance sheet. The image quality score is obtained by performing a sharpness calculation on the image region corresponding to the target cell. The sharpness calculation includes a sharpness assessment based on the Laplacian operator or the gradient operator. The comprehensive score is a weighted combination of OCR confidence, image quality score, and residual penalty term. The residual penalty term is penalized when the residual of the accounting formula involving the target cell is greater than zero, and is not penalized when the residual of the accounting formula is zero. The result selection or fusion includes: comparing the cloud recognition result with the local recognition result, and preferentially selecting the value that reduces the residual of at least one accounting formula involving the target cell the most as the final value; when multiple candidate cells jointly affect the same accounting formula, the final values ​​of multiple candidate cells are jointly selected or iteratively updated according to the residual minimization criterion.

[0037] This invention also provides a financial report image data extraction and verification system, deployed on one or more computing devices, characterized in that it includes: The document interface module is used to receive financial report image inputs and output structured financial data and a manual review interface; The preprocessing module is used to perform sensitive information area detection locally to obtain the seal area mask and signature area mask, and to perform desensitization processing on the financial report image based on the signature area mask. At the same time, it performs seal removal processing on the desensitized financial report image based on the seal area mask to generate a seal-removed financial report image. The hybrid OCR module includes a cloud-based OCR interface and a local OCR engine. The cloud-based OCR interface is used to recognize the image of the financial report without the seal and output the cloud recognition result, which includes at least the cell recognition text, cell coordinate information and OCR confidence. The local OCR engine is used to backtrack and re-recognize the high-definition slice of the original un-de-sensitized image when the comprehensive score is lower than the threshold. The accounting verification module includes an accounting formula rule library, which is used to execute accounting formula calculations and output accounting formula residuals, and to calculate a comprehensive score to trigger local rollback and re-identification; The accounting subject standardization module is used to map the identified accounting subject text to a standard accounting subject library and generate structured financial data; The data storage module is used to store the standard accounting subject library, accounting formula rule library, mapping rules, and structured financial data and verification results.

[0038] The accounting subject standardization module includes a synonym extension set construction unit and a dual similarity calculation unit. The dual similarity calculation unit is used to calculate the literal similarity and semantic similarity between the accounting subject text and the standard accounting subject, and then weight the two to obtain a mapping score. When the mapping score is lower than the subject mapping threshold, the accounting subject text is marked as an item to be manually confirmed and pushed to the manual review interface.

[0039] It also includes an accounting expert intelligent agent module, which is a rule-based expert system; when the accounting verification module reports that the accounting formula calculation has failed, the accounting expert intelligent agent module traces back the accounting subject data related to the failed formula based on the accounting rule knowledge base, locates one or more numerical cells that are suspected of being identified incorrectly, and generates repair suggestions for manual review.

[0040] It also includes a Trusted Execution Environment (TEE) module and a formal constraint solver module; The Trusted Execution Environment (TEE) module is used to perform sensitive information region detection, desensitization, and seal removal processing of the preprocessing module within the Trusted Execution Environment; high-definition slice cropping of the original undesensitized image required for local rollback and re-identification of the hybrid OCR module; and local OCR inference. Before sending the seal-removed financial report image to the cloud, it generates a remote verification report bound to the metric value and program version of the Trusted Execution Environment. The cloud OCR interface in the hybrid OCR module only receives the seal-removed financial report image and returns the cloud recognition result when the remote verification report is verified successfully. The formal constraint solver module is used to formalize the accounting formula rule base in the accounting verification module into a constraint set, and construct a candidate value set for discrete variables by combining the cloud recognition results and the candidate values ​​output by the local OCR engine. An optimization model is established with the objective function of minimizing the accounting formula residual and minimizing the number of replaced cells and / or the replacement cost. The optimal solution that satisfies the constraint set is obtained by solving the model using Satisfiability Modulo Theories (SMT) or integer linear programming, so as to determine the final value of each target cell for outputting structured financial data.

[0041] It also includes a digital forensics anti-forgery detection module, which is configured to perform tampering / forgery risk analysis on the financial report image before or after the preprocessing module outputs the image of the financial report with the seal removed, including at least one or more of the following detections: compression consistency detection based on error level analysis (ELA), compression parameter consistency detection based on JPEG quantization table / block effect, consistency detection based on image noise fingerprint (PRNU), and splicing anomaly detection based on layout geometric consistency and font / character spacing statistical features, and output an evidence risk score R_forensic; Specifically, when calculating the comprehensive score, the accounting verification module incorporates the evidence collection risk score R_forensic as a risk penalty item, so that the comprehensive score simultaneously reflects OCR confidence, image quality score, accounting formula residuals, and evidence collection risk. Furthermore, when the evidence collection risk score R_forensic reaches or exceeds a preset evidence collection threshold, at least one handling strategy is triggered: performing local rollback re-identification on the target cell related to the accounting formula residual, marking the corresponding structured financial data as pending manual review, and / or outputting tampering risk warning information, thereby further suppressing the risk of erroneous data entry caused by forged or tampered images on the basis of closed-loop verification of accounting residuals.

[0042] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a method for extracting and verifying financial report image data.

[0043] Example 1: Data Extraction and Verification of Financial Report Images I. Implementation Environment and Inputs / Outputs 1. Computing devices (client / local side) CPU: 4 cores or more; RAM: ≥8GB; SSD: ≥50GB; Local OCR inference supports: GPU (≥4GB VRAM) or pure CPU inference; Operating system: Windows / Linux; Network: Accessible via HTTPS interface to cloud-based OCR servers.

[0044] 2. Cloud side Cloud-based OCR server: Provides APIs for table detection, cell segmentation, character recognition, and confidence score output; the returned results include the recognized text value and character confidence score for each cell.

[0045] 3. Input Financial report images: PNG / JPG or bitmaps rendered page by page from a PDF; Recommended image resolution: longer side ≥ 2000 pixels; or scan DPI ≥ 200; Typical elements include: red stamps, signatures, table lines, subject text, and numerical values.

[0046] 4. Output Structured financial data (JSON / tabular are both acceptable), with fields including at least: report type, period, standard accounting subject, ending / beginning balance, confidence level, and validation flags; Verification results: Accounting formula residual Δ, list of abnormal cells, whether manual confirmation is required, and expert agent repair suggestions (if triggered).

[0047] II. Key Configuration Parameters This embodiment provides a set of recommended parameters: 1. Sensitive Information Area Detection Model Parameters Model: Improved YOLO or Mask R-CNN; used to locate sensitive information regions (stamps, signatures, etc.).

[0048] Confidence threshold: 0.60; NMS threshold: 0.45; Mask output: Pixel-level masks include at least a stamp area mask M_stamp and a signature area mask M_sign (each mask is a binary value of 0 / 1).

[0049] 2. Desensitization and stamp removal parameters (local execution) Desensitization method A: Apply Gaussian blur to the signature area, kernel size = 31×31, σ = 7; Desensitization method B: Zero-padding is performed on the signature area; Stamp Removal: Perform image restoration (StampRemoval) on the stamp area.

[0050] Important constraint: The original, un-de-identified images are stored only in local memory / local controlled storage and are not transmitted over the network.

[0051] 3. Accounting Logic Rules and Residual Calculation Accounting identity (example of balance sheet): Total assets = Total liabilities + Total owner's equity; Residual formula: Δ = |Value on the left side of the formula Value on the right side of the formula|; If Δ = 0, it passes; if Δ > 0, it fails.

[0052] 4. Image quality score and comprehensive scoring model Image quality score Q_img: Normalized to [0,1] using Laplacian variance of sharpness; In this embodiment: If Var(Laplace) ≥ 150, it is recorded as clear (Q_img ≈ 1); if ≤ 30, it is recorded as blurred (Q_img ≈ 0), and linearly interpolated for normalization.

[0053] Residual penalty term λ(Δ): If there is a residual in the formula that the cell participates in, take 1; otherwise, take 0; Weight coefficients: w1 = 0.55, w2 = 0.35, w3 = 0.10; Calculation of comprehensive score R_i (fixed adoption in this embodiment): R_i = w1·C_ocr + w2·Q_img w3·λ(Δ) Where C_ocr is the confidence of the cloud OCR in the value of this cell.

[0054] 5. Local fallback re-identification threshold and local OCR Scoring threshold T: 0.75; If R_i ≥ T, adopt the cloud result; if R_i < T, trigger local fallback re-identification.

[0055] Local fallback cropping: Crop high-definition slices from the local original un-desensitized image according to the cell coordinates returned by the cloud; Local OCR engine: MobileNet-CRNN lightweight model; Digital character set "0-9, comma, decimal point, negative sign".

[0056] Result selection rule: Prioritize selecting the value that can make the formula residual Δ smaller or zero as the final result.

[0057] 6. Accounting subject normalization threshold Standard accounting subject library: Each standard subject is associated with a synonym / synonymous extension set; Literal similarity: Edit distance (Levenshtein Distance) converted to similarity s_lex = 1 dist / maxlen; Semantic similarity: Cosine similarity of text vectors s_sem; Weighted score: S = 0.6·s_sem + 0.4·s_lex; Mapping threshold: 0.82; Items below the threshold are marked as "items requiring manual confirmation".

[0058] III. Implementation Process: Step S101: Receive financial report image The client-side document interface module receives the financial report image I_raw (e.g., a balance sheet page) to be recognized. If the input is a PDF, it first renders each page as a bitmap and records the page number and rendering resolution.

[0059] Step S102: Local Sensitive Information Area Detection and Desensitization 1. Call the sensitive information region detection model to perform inference on I_raw and output the sensitive region bounding box / segmentation mask, where the sensitive information region includes at least the red seal region and the legal representative / legal representative signature region.

[0060] 2. After generating the pixel-level mask, perform Gaussian blur (kernel 31×31, σ=7) on the mask region to obtain the desensitized image I_des; 3. To meet financial compliance constraints, I_raw is stored only in local memory / local controlled storage and is not transmitted over the network; only I_des is used on the network side.

[0061] Step S103: Stamp Removal (Image Restoration) The stamp removal algorithm is applied to I_des, outputting the stamp-removed image I_sr (StampRemoved). This processing addresses the case where a stamp obscures text / numbers, attempting to restore the obscured strokes through repair methods, thereby improving the readability of subsequent OCR.

[0062] Step S104: Initial cloud-based OCR recognition and confidence level acquisition Send the I_sr to the cloud OCR server via HTTPS: Perform text detection, table structure parsing, cell segmentation, and character recognition in the cloud; Returns a structured result D_cloud, which contains the recognized text value_i, cell coordinates bbox_i, and character / cell confidence score C_ocr(i) for each cell.

[0063] Step S105: Determining the Report Type and Calculating Residuals of Accounting Formulas 1. Determine the report type based on the title / header keywords in D_cloud (balance sheet in this example); 2. Load the set of accounting equations (which must include at least "Total Assets = Total Liabilities + Total Owner's Equity"); 3. Extract the corresponding values ​​from D_cloud and calculate the residuals: Δ = |Total Assets (Total Liabilities + Total Owner's Equity)|; 4. Record Δ and the set of key cells K = {Total Assets cell, Total Liabilities cell, Total Owner's Equity cell} participating in this identity.

[0064] Step S106: Multidimensional Comprehensive Scoring and Threshold Trigger Judgment Calculate the comprehensive score R_i for each numeric cell one by one: 1. Calculate Q_img(i): Crop the cell region Patch_i (by bbox_i) from I_sr, calculate Var(Laplace(Patch_i)) and normalize it to [0, 1]; 2. Calculate λ(Δ): If cell i belongs to K and Δ > 0, then λ(Δ) = 1; otherwise, λ(Δ) = 0; 3. Calculate R_i = 0.55·C_ocr(i) + 0.35·Q_img(i) 0.10·λ(Δ); 4. If R_i ≥ T(0.75), then tentatively adopt the cloud result value_i; if R_i < T, then mark it as the candidate cell set F that needs to be re-recognized locally with fallback.

[0065] Step S107: Local Fallback Re-Recognition and Result Fusion (Key Error Correction Closed Loop) Execute for each candidate cell i in set F: 1. Crop the high-definition slice Patch_raw(i) from the local original un-desensitized image I_raw according to bbox_i to retain details that may be caused by desensitization / blurring; 2. Call the local lightweight OCR (MobileNet-CRNN) to recognize Patch_raw(i), obtaining the local recognition value value_local(i) and the local confidence C_local(i); 3. Result fusion: Select the final value value_final(i) between value_i and value_local(i) with the criterion of "residual minimization first" to reduce or zero Δ; 4. If multiple candidate cells jointly affect the same identity, then iterate 1 - 3 rounds in the way of "replace cells one by one - recalculate Δ - select the combination with the smallest Δ" until Δ = 0 or the iteration upper limit is reached.

[0066] Balance Sheet Identity Error Correction: Cloud-based OCR recognition: Total assets = 1,000,800 (C_ocr = 0.93), Total liabilities = 600,500 (C_ocr = 0.92), Total owner's equity = 400,200 (C_ocr = 0.51, and the cell has low clarity Q_img = 0.30).

[0067] Calculate Δ = |1,000,800 (600,500+400,200)|=100 (Δ>0).

[0068] For the cell "Total Owner's Equity", calculate R_i = 0.55 × 0.51 + 0.35 × 0.30. 0.10 × 1 = 0.2805 + 0.105 0.10 = 0.2855 < 0.75, triggering a local rollback.

[0069] Local OCR recognition value_local=400,300; after replacement Δ'=|1,000,800 (600,500+400,300)|=0, therefore 400,300 is chosen as the final value to complete the error correction loop.

[0070] Step S108: Standardize accounting subjects (unify terminology for easier data entry and analysis) Perform normalization on the subject text in D_cloud / D_final: 1. Pre-build a standard accounting subject library, with each standard subject associated with an extended set of synonyms / near-synonyms; 2. Calculate the following for each subject's original SourceText extracted by OCR: Literal similarity s_lex (based on edit distance); Semantic similarity s_sem (based on text vector cosine similarity); 3. Calculate the overall score S = 0.6·s_sem + 0.4·s_lex, and select the standard subject with the highest S; if S < 0.82, mark it as "item to be manually confirmed" and output the top 3 candidate standard subjects for review.

[0071] Step S109: Accounting expert AI agent assists in repair When the accounting formula calculation engine still reports a verification failure (e.g., "details summary does not equal total"), the accounting expert agent is activated: This agent is a rule-based expert system that automatically backtracks the subject data related to the failed formula based on a preset accounting rule knowledge base, locates the numerical cells suspected of having OCR errors, and generates repair suggestions (e.g., "prioritize reviewing the 'operating profit' ending balance or its detailed items") for manual verification.

[0072] Step S110: Perform integrity and consistency checks and output structured results. 1. Integrity check: Check if key fields are missing (report type, period, total items, etc.); 2. Consistency check: Recalculate the balance and reconciliation relationships within the table, and output the final residual Δ_final; 3. Output structured financial data (Data_out) and a verification report (Report_out).

[0073] IV. Structured Output Example: Data_out: report_type: Balance Sheet; period: 2024; items: [{std_subject: "Cash and Funds", end_value: ..., begin_value: ..., source_bbox: ..., source_conf: ..., final_conf: ..., need_manual_confirm: 0 / 1}, ...] check: {delta_balance_sheet: Δ_final, fallback_count: number of fallbacks triggered, manual_confirm_count: number of items awaiting manual confirmation} expert_suggestion: [{rule_id: ...,suspect_cells: [...],suggestion: "..."}] (if triggered) Example 2: I. System Structure (Modules, Connections, and Data Flow) 1. Trusted Execution Environment (TEE) module (local side) It consists of TEE hardware / firmware (e.g., any one of Intel SGX, AMD SEV-SNP, ARM TrustZone) and TEE runtime; Trusted code runs within the TEE, including "sensitive region detection, desensitization, high-resolution original image cropping, local OCR inference, candidate set construction, formal solution and result selection"; The TEE only runs a host process (HostApp) for I / O, network communication, and UI display. The host process does not access the original, un-de-identified plaintext images.

[0074] 2. Remote Attestation Component (Local Side + Cloud Side) Local side: The TEE generates a remote attestation report, which is at least bound with: trusted code measurement value (Measurement), version number, random number Nonce, and timestamp. Cloud side: The verification service verifies the signature and policy of the attestation report, and only allows the cloud OCR interface to accept requests after passing the verification ("attest first, then serve").

[0075] 3. Formal constraint solver module (local side, recommended to be executed within the TEE) Input: Accounting formula rule library (identities / arithmetical relations), cloud OCR result candidates, local fallback OCR candidates, confidence and quality scores of each candidate. Output: The final value of each target cell, which minimizes the overall accounting formula residual, and when feasible, achieves Δ zeroing, while minimizing the replacement cost / number of replaced cells.

[0076] II. Key parameter configuration: This embodiment adopts the following parameters: 1) TEE and remote attestation parameters TEE type: Intel SGX (example), the enclave measurement value field is denoted as MRENCLAVE; Validity period of the attestation report: TTL_attest = 300 seconds (5 minutes); Length of the random number Nonce: 16 bytes, generated and distributed by the cloud; Attestation passing policy: MRENCLAVE ∈ whitelist, version number ≥ minimum security version, and timestamp within TTL_attest; Network channel: TLS1.2 / 1.3, certificate verification enabled on the cloud; mutual TLS is enabled to strengthen identity authentication.

[0077] 2) Desensitization and OCR basic parameters Sensitive area detection: Pre-trained object detection / segmentation model (such as improved YOLO or Mask R-CNN), outputting a pixel-level mask; sensitive areas at least include red seals, legal person / auditor signature areas; Desensitization method: Gaussian blur or pixel zero filling; the original non-desensitized image is only saved in local memory and not transmitted over the network; Cloud OCR return: Cell recognition text and character / cell confidence C_ocr; Accounting identity: Total assets = Total liabilities + Total owner's equity; the residual Δ is calculated by absolute value; Multi-dimensional scoring and fallback: If R_i < T triggers local fallback re-identification, the original image is cropped into high-definition slices and re-identified using local lightweight OCR, and the value that makes Δ smaller or zero is preferentially selected; 3) Candidate set and formal solution parameters (new in this embodiment) Scaling factor for amount: S=100 (in cents to avoid floating-point errors; for example, 1,234.56 yuan → 123456). Number of candidate values ​​per cell: K=2 (1 candidate in the cloud + 1 candidate for local rollback; if local rollback is not triggered, only cloud candidate is considered). The set of cells included in the solution, U, consists of all numeric cells that participate in any accounting formula / relationship (including at least total items and key relationship items). Objective function weights: Residual weights: α = 1.0 Replacement cost weight: β=0.2 (preferring fewer changes and prioritizing higher confidence levels) Replacement quantity penalty: γ=50 (unit: points; used for strong preferences to not replace or replace less). III. Implementation Steps: Step S201: Receive the original financial report image and load it into the TEE. The host process receives the raw image I_raw (PNG / JPG or PDF rendered bitmap) and only passes it into the TEE via encrypted shared memory or a TEE-controlled buffer. The raw image plaintext is not stored outside the TEE.

[0078] Step S202: Perform sensitive region detection and generate a mask within the TEE. TEE internally calls the sensitive region detection model to infer sensitive region mask from I_raw. Sensitive regions include, but are not limited to, the red stamp and signature areas.

[0079] Step S203: Complete desensitization within the TEE and generate a desensitized image. Within the TEE, a Gaussian blur (or pixel zero-filling) is applied to the masked area to obtain the desensitized image I_des. Meanwhile, the TEE retains I_raw only for subsequent local rollback cropping if necessary; and adheres to the compliance constraint that "the original undesensitized image is only stored in local memory and not transmitted over the network."

[0080] Step S204: Perform stamp removal on the desensitized image within the TEE to obtain a stamp-free image. The stamp removal algorithm is applied to I_des to obtain I_sr (stamp-removed image) to improve the problem of missing characters / numbers caused by stamp occlusion.

[0081] Step S205: The TEE generates a remote verification report and establishes a trusted session with the cloud. 1. The cloud verification service issues a random number (Nonce). 2. The TEE generates an attestation report: AttestReport = Sign_TEE(MRENCLAVE, version number, Nonce, timestamp); 3. The host process sends AttestReport to the cloud verification service; 4. After successful cloud verification, a SessionToken is returned (valid for 300 seconds in this embodiment) for subsequent authentication of cloud OCR requests.

[0082] Step S206: Upload the de-stamped and desensitized image and obtain the cloud OCR structured result Send I_sr (desensitized and de-stamped) along with SessionToken to the cloud OCR server. The cloud performs text detection, table structure parsing, and cell recognition, and returns the structured result D_cloud, which includes the recognized text value and confidence level C_ocr for each cell.

[0083] Step S207: Load the accounting formula rule library in the TEE and calculate the initial residual Δ Load the set of accounting identities according to the recognized report type. For example, for the balance sheet, "Total assets = Total liabilities + Total owner's equity", and calculate the residual: Δ = |Value on the left side of the formula Value on the right side of the formula|. If Δ = 0, it passes; if Δ > 0, it fails.

[0084] Step S208: Calculate the multi-dimensional score in the TEE and trigger local fallback re-identification (generate a candidate set) 1. Calculate the comprehensive score R_i for each numeric cell (including C_ocr, image quality score Q_img (Laplacian sharpness), and residual penalty term); 2. Set a threshold T. If R_i < T, trigger local fallback: Crop a high-definition slice Patch_raw(i) from I_raw in the TEE according to the cell coordinates returned by the cloud, and call the local lightweight OCR (MobileNet-CRNN) to obtain the local recognized value v_local(i); 3. Construct the candidate set: Cloud candidate: v_cloud(i) (from D_cloud) Local candidate: v_local(i) (only exists when fallback is triggered) Form Cand(i) = {v_cloud(i)} or {v_cloud(i), v_local(i)} (K ≤ 2 in this embodiment) Step S209: Formalize the accounting rules into a constraint set and construct discrete variables In the TEE, uniformly convert the amount to an integer in "cents": val(i,k)=round(v_{i,k}S), S=100; For each cell i included in the solution set U, establish a discrete selection variable x_{i,k}∈{0,1} to indicate whether to select candidate k; Constraint: For each i, Σ_kx_{i,k}=1 is satisfied (each cell must select one and only one candidate value).

[0085] Step S210: Establish the objective function (minimum residual + minimum replacement cost + minimal modifications) For each accounting formula e (such as the asset identity), establish a residual variable r_e≥0, and introduce an absolute value linearization constraint: Let L_e and R_e be the values ​​obtained by linear combinations of cells on the left and right sides of the formula, respectively (both are integers in "cents"); Constraint: r_e≥(L_e) R_e) and r_e≥ (L_e R_e); Objective function: Minimizeα·Σ_er_e+β·Σ_iΣ_kCost(i,k)·x_{i,k}+γ·Σ_iReplace(i) in: Cost(i,k) is the candidate cost, and Cost(i,k) = round((1) Conf(i,k))·100) (Conf comes from the fusion of cloud / local OCR confidence and quality score); Replace(i) indicates whether a replacement occurs (e.g., Replace=1 if a local candidate is selected, otherwise 0), to encourage "minimizing changes".

[0086] Step S211: Call the solver to obtain the optimal solution and generate the final structured numerical solution. The solver (MILP / SMT engine) is called within the TEE to obtain the optimal solution {x_{i,k}}, thus obtaining the final value of each cell: val_final(i)=Σ_kval(i,k)·x_{i,k}, And convert back to the original value: v_final(i) = val_final(i) / S.

[0087] Subsequently, v_final(i) is used to recalculate the residuals of all formulas to obtain Δ_final, and a verification report is generated.

[0088] Step S212: Output Results and Credible Evidence Output structured financial data (including standard accounts, ending / beginning balances, confidence levels, whether replacements are needed, participation formulas, etc.), and: Δ_final, each formula r_e, and a list of cells to be replaced; Remote proof digests (e.g., MRENCLAVE and proof pass time) are used to prove that the current process was executed in trusted code.

[0089] IV. Numerical Examples: Taking the balance sheet identity as an example, cloud-based OCR and local rollback offer the following candidates (unit: yuan): Total assets: Cand(A) = {1,000,800} (Cloud, high R, no rollback triggered) Total liabilities: Cand(L) = {600, 500} (Cloud-based, high R, no rollback triggered) Total Owner's Equity: Cand(E) = {400, 200 (Cloud), 400, 300 (Local Rollback)} (Low confidence in the cloud triggers rollback) Formal model constraint: A = L + E.

[0090] If we choose cloud-based E=400,200, then the residual Δ=|1,000,800 (600,500+400,200)|=100 (not equal to 0); If the local value E=400,300 is selected, then the residual Δ=0.

[0091] With α much greater than β and γ, the solver will choose E=400,300 to bring the residual to zero and only modify one cell to achieve the global optimal consistency repair of "minimum Δ + minimum modification".

[0092] Example 3: I. Implementation Environment and Inputs / Outputs 1. Local computing device (client) CPU: 4 cores or more; RAM: ≥8GB; SSD: ≥50GB; GPU: ≥4GB video memory (used to accelerate forensics and local OCR reasoning); Operating environment: Windows / Linux; Network: HTTPS access to the cloud OCR server (upload only de-identified images).

[0093] 2. Input Financial report image I_raw: PNG / JPG, or a bitmap rendered page by page in PDF; Image resolution recommendation: longer side ≥ 2000 pixels (for easier table line / font statistics and PRNU calculation).

[0094] 3. Output Structured financial data Data_out (including subject, end / beginning period, confidence level, verification flag); Verification report Report_out: accounting residual Δ, reasons for triggering rollback / manual review, forensic risk score R_forensic, and risk warnings.

[0095] II. System Structure (Adding a Digital Forensics Module to the Basic System) This embodiment newly adds and enables: 1. Digital forensics anti-forgery detection module (executed on the local side) Input: I_raw (raw image) and / or I_des (desensitized image) and / or I_sr (de-stamped image); Output: R_forensic ∈ [0, 1] (the larger the value, the higher the tampering / forgery risk) and forensic evidence items (e.g., ELA abnormal hot areas, PRNU inconsistent areas, layout geometric anomaly indicators, etc.).

[0096] 2. Enhancement of Comprehensive Scoring and Trigger Strategy Introduce R_forensic as a risk penalty term into the cell comprehensive scoring; and set the forensic threshold trigger strategy: trigger local rollback re-identification / mark manual review / output risk warnings.

[0097] Steps of the basic process: local sensitive area detection and desensitization, cloud OCR confidence level acquisition, calculation of accounting identity residual Δ, multi-dimensional scoring R_i, local rollback re-identification, and result selection of "preferably choosing the value that makes Δ smaller or zero".

[0098] III. Key Parameters: 1) Basic Parameters Local constraint for the original non-desensitized image: I_raw is only stored in local memory and not transmitted over the network.

[0099] Cloud OCR output: recognized text and confidence level C_ocr for each cell.

[0100] Accounting residual: Δ = |value on the left side of the formula value on the right side of the formula|; Δ = 0 passes, Δ > 0 fails.

[0101] Local rollback trigger: scoring threshold T; if R_i < T, then cut a high-definition slice from I_raw and perform secondary local OCR recognition, preferably choosing the value that makes Δ smaller or zero.

[0102] 2) Digital Forensics Parameters (Newly Added in This Embodiment) This embodiment adopts "4 types of forensic detections + weighted fusion": (a) ELA compression consistency test If the input is JPEG: use the original image directly; if it is PNG / bitmap: first recompress it with a JPEG quality factor of Q=90 to generate I_q90. Calculate the ELA difference plot: E=|I I_q90| (absolute difference per pixel); Metric: s_ela=clamp((mean(E)) μ0) / (μ1 μ0),0,1), where μ0=2.0 and μ1=12.0 (units of grayscale difference); (b) JPEG quantization / blocking consistency detection If it is JPEG: extract the quantization table and calculate the energy distribution consistency of the 8×8 block DCT; If not JPEG: Calculate the block effect index B (mean difference of gradients of adjacent 8 pixels) and normalize it to s_block∈[0,1]; Threshold: B ≥ 0.18 is considered a block effect anomaly (corresponding to s_block close to 1).

[0103] (c) PRNU noise fingerprint consistency detection (segmentation detection) Divide the page into m×n=6×4 non-overlapping grid blocks; Denoising is performed on each block (wavelet / guided filtering) to obtain the noise residual W_{i}; Calculate the normalized cross-correlation NCC(W_i, W_j) between adjacent blocks, and ensure global consistency: c_prnu=mean(NCC(W_i,W_j)) (adjacency pairs); Risk score: s_prnu=clamp((c0 c_prnu) / (c0 c1),0,1), where c0=0.12 and c1=0.02.

[0104] This PRNU consistency check is applicable to scenarios where images are taken / scanned and then stitched together. If the image undergoes strong noise reduction or resampling, resulting in an overall low NCC, the system will automatically reduce the weight of the PRNU item (see the fusion weight adaptive rule).

[0105] (d) Anomaly detection in page layout geometry and font / space statistics Table line detection: The main direction angle θ is obtained using Hough line detection; Text line baseline detection: Projecting the text region to obtain the line direction angle φ; Directional consistency index: d_ang = |θ φ (degrees); Character spacing statistics: Extract the character bounding boxes for each line of text and obtain the coefficient of variation CV_gap of the spacing sequence; Risk Score: s_layout=clamp(d_ang / 3.0,0,1)0.6+clamp((CV_gap 0.35) / (0.80 0.35),0,1)0.4.

[0106] (When d_ang≥3° or CV_gap≥0.80, it is considered that there is a strong suspicion of splicing / deformation.) The total forensic risk score is obtained by fusion. Weights (fixed in this embodiment): w_ela=0.30,w_block=0.20,w_prnu=0.25,w_layout=0.25; R_forensic=w_elas_ela+w_blocks_block+w_prnus_prnu+w_layouts_layout; Evidence collection threshold: The warning threshold T_forensic_warn = 0.60; The mandatory review threshold T_forensic_hard = 0.80.

[0107] IV. Implementation Steps: Step S301: Receive the original report image The client receives I_raw (raw financial report image) for local processing and possible subsequent local rollback cropping only, and does not transmit it over the network.

[0108] Step S302: Perform sensitive area detection and desensitization locally. A pre-trained object detection model is used to locate sensitive information regions (such as seals and signatures), generate pixel-level masks, and perform Gaussian blur / zero-filling to obtain the desensitized image I_des.

[0109] Step S303: Perform stamp removal to obtain the stamp-removed image. Perform the StampRemoval algorithm on I_des to generate the stamp-removed image I_sr.

[0110] Step S304: Local digital forensics anti-forgery detection and output R_forensic The digital forensics module performs the above (a) to (d) checks on I_raw (preferred, as the information is more complete) to obtain s_ela, s_block, s_prnu, and s_layout and merges them into R_forensic. At the same time, it records evidence items (such as ELA abnormal area heatmap coordinates, PRNU abnormal grid block numbers, etc.).

[0111] Step S305: Upload the de-identified image (with stamp removed) to the cloud-based OCR system and obtain the confidence level. I_sr is sent to a cloud-based OCR server for recognition, resulting in a structured result D_cloud, which contains the recognized text and confidence score C_ocr for each cell.

[0112] Step S306: Load the set of accounting identities and calculate the accounting residual Δ Load the accounting identity / reconciliation rule set (e.g., balance sheet identity) based on the identified report type, substitute the numerical values ​​to calculate the residual Δ, and obtain the verification status of Δ=0 / Δ>0.

[0113] Step S307: Construct a comprehensive cell score including "evidence collection and penalty items". Calculate the comprehensive score R_i' for each numerical cell i (adding a penalty item based on evidence collection to R_i): Definitions: C_ocr, Q_img (Laplace sharpness), λ(Δ) (residual penalty term), and weighting coefficients.

[0114] This embodiment adds a new evidence collection penalty item: R_i'=w1C_ocr(i)+w2Q_img(i) w3λ_i(Δ) w4R_forensic Where w4 = 0.15, and the remaining w1, w2, w3 are the same as the engineering values ​​given in Example 1.

[0115] Step S308: Evidence collection threshold triggering strategy (linked with accounting residual closed loop) If R_forensic ≥ T_forensic_hard (0.80): 1. Mark the entire page of results as "Forced Manual Review"; 2. Output a high-risk warning for tampering; 3. Trigger local rollback and re-identification for all key cell sets K involved in any accounting identity / reconciliation rule (see S309).

[0116] If T_forensic_warn(0.60)≤R_forensic <T_forensic_hard(0.80): 1. If there exists Δ > 0, then only trigger local fallback re-identification for the set of cells K related to Δ; 2. At the same time, add the cells covered by the ELA / PRNU abnormal area to the fallback set F; 3. Output a "medium risk" prompt and an evidence item summary.

[0117] If R_forensic < T_forensic_warn(0.60): Trigger fallback only when R_i' < T.

[0118] Step S309: Local fallback re-identification and result selection For the cells in the fallback set, the system cuts high-definition slices from the local I_raw according to the cell coordinates returned by the cloud, calls the local lightweight OCR for secondary identification, and compares between the cloud result and the local result, and preferentially selects the value that can make the accounting residual Δ smaller or zero as the final result.

[0119] Step S310: Perform subject normalization and pending confirmation marking if necessary Perform standard accounting subject library mapping (synonym expansion set + literal / semantic double similarity) on the subject text. If it is below the threshold, mark it as an "item to be manually confirmed" and enter the review interface.

[0120] Step S311: Output structured data and verification / forensic report Output Data_out and Report_out, where Report_out at least includes: R_forensic, trigger level (low / medium / high risk), forensic evidence items; Accounting residual Δ and whether it is zeroed, list of cells that trigger fallback; Reasons for whether manual review is required ("forensic high risk / accounting imbalance / low confidence / image blur", etc.).

[0121] V. Example (How Forensic Score Affects Closed-Loop Error Correction) Suppose the key total items obtained by cloud OCR on the balance sheet page are: Total assets = 1,000,800 (C_ocr = 0.93) Total liabilities = 600,500 (C_ocr = 0.92) Total owner's equity = 400,200 (C_ocr = 0.51) Then the accounting residual Δ = |1,000,800 - (600,500 + 400,200)| = 100 > 0.

[0122] At the same time, the digital forensic module outputs: s_ela=0.70,s_prnu=0.65,s_layout=0.55,s_block=0.20 R_forensic = 0.30 + 0.70 + 0.25 + 0.65 + 0.25 + 0.55 + 0.20 = 0.56 (below the warning threshold of 0.60) At this point, the system still triggers rollback based on accounting residual closure and cell scoring; it triggers local rollback re-identification for "Total Owner's Equity", resulting in local OCR=400,300. After replacement, Δ is set to zero, the final value is output, and "rollback triggered because Δ>0" is recorded.

[0123] If the evidence output on another page is R_forensic=0.83 (≥0.80), even if Δ is temporarily 0, it will be marked as requiring manual review and a high-risk warning will be output. At the same time, key cross-reference items will be rolled back for re-identification and evidence item display to reduce the risk of "forgery but cross-reference superficially valid".

[0124] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made under the inventive concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of the present invention.

Claims

1. A method for extracting and verifying image data from financial reports, characterized in that, include: S1. Obtain the financial report image to be processed, and perform sensitive information region detection on the financial report image locally to obtain a pixel-level mask of the sensitive information region. The pixel-level mask includes at least a seal region mask and a signature region mask. S2. Perform desensitization and seal removal processing on the financial report image based on the pixel-level mask: Perform desensitization processing on the financial report image based on the signature area mask to generate a desensitized financial report image; The desensitized financial report image is then subjected to stamp removal processing based on the stamp area mask to generate a stamp-removed financial report image; wherein, the original undesensitized financial report image is only kept locally and is not transmitted over the network; S3. Send the image of the financial report without seal to the cloud OCR server for recognition and obtain the cloud recognition result. The cloud recognition result includes at least the recognized text of the table cell, cell coordinate information and the corresponding OCR confidence score. S4. Determine the report type based on the cloud recognition result and load the accounting formula rule set corresponding to the report type. Perform accounting formula calculation on the values ​​in the cloud recognition result to obtain at least one accounting formula residual. S5. For at least one numerical cell, calculate the comprehensive score of the numerical cell based on the OCR confidence of the numerical cell, the image quality score of the numerical cell, and the residual penalty term corresponding to the accounting formula residual, and compare the comprehensive score with a preset score threshold. S6. When the overall score is lower than the preset score threshold, trigger local rollback and re-identification: crop a high-definition slice corresponding to the target cell from the original un-anonymized financial report image, call the local OCR engine to perform secondary recognition on the high-definition slice to obtain the local recognition result, and select or fuse the results of the local recognition result and the cloud recognition result based on the residual minimization criterion, and output the final value of the target cell; when the overall score is not lower than the preset score threshold, use the cloud recognition result as the final value of the target cell; S7. Perform accounting subject normalization on the identified accounting subject text: map the accounting subject text to the standard accounting subjects in the standard accounting subject library; and after completing the accounting subject normalization, perform integrity verification and accounting formula calculation verification on the structured financial data, and output the structured financial data and verification results.

2. The method according to claim 1, characterized in that, The sensitive information region detection uses a pre-trained target detection model to scan and locate the financial report image and generate the pixel-level mask. The pixel-level mask includes at least a seal region mask and a signature region mask. The desensitization process includes performing Gaussian blur processing and / or zero-filling processing on the image region corresponding to the signature region mask.

3. The method according to claim 1, characterized in that, The stamp removal process includes performing image inpainting and reconstruction on the stamp-occluded area to restore the obscured character or number stroke information; the image inpainting and reconstruction is implemented using a generative adversarial network model and / or a content-aware image inpainting model.

4. The method according to claim 1, characterized in that: The accounting formula rule set includes at least intra-statement balance verification rules and cross-statement reconciliation relationship verification rules; wherein, the intra-statement balance verification rules include at least the verification of the identity relationship between total assets, total liabilities and total owners' equity, and the cross-statement reconciliation relationship verification rules include at least the verification of the reconciliation relationship between net profit in the income statement and retained earnings in the balance sheet. The image quality score is obtained by performing a sharpness calculation on the image region corresponding to the target cell. The sharpness calculation includes a sharpness assessment based on the Laplacian operator or the gradient operator. The comprehensive score is a weighted combination of OCR confidence, image quality score, and residual penalty term. The residual penalty term is penalized when the residual of the accounting formula involving the target cell is greater than zero, and is not penalized when the residual of the accounting formula is zero. The result selection or fusion includes: comparing the cloud recognition result with the local recognition result, and preferentially selecting the value that reduces the residual of at least one accounting formula involving the target cell the most as the final value; when multiple candidate cells jointly affect the same accounting formula, the final values ​​of multiple candidate cells are jointly selected or iteratively updated according to the residual minimization criterion.

5. A financial report image data extraction and verification system, deployed on one or more computing devices, for implementing the method according to any one of claims 1-4, characterized in that, include: The document interface module is used to receive financial report image inputs and output structured financial data and a manual review interface; The preprocessing module is used to perform sensitive information area detection locally to obtain the seal area mask and signature area mask, and to perform desensitization processing on the financial report image based on the signature area mask. At the same time, it performs seal removal processing on the desensitized financial report image based on the seal area mask to generate a seal-removed financial report image. The hybrid OCR module includes a cloud-based OCR interface and a local OCR engine. The cloud-based OCR interface is used to recognize the image of the financial report without the seal and output the cloud recognition result, which includes at least the cell recognition text, cell coordinate information and OCR confidence. The local OCR engine is used to backtrack and re-recognize the high-definition slice of the original un-de-sensitized image when the comprehensive score is lower than the threshold. The accounting verification module includes an accounting formula rule library, which is used to execute accounting formula calculations and output accounting formula residuals, and to calculate a comprehensive score to trigger local rollback and re-identification; The accounting subject standardization module is used to map the identified accounting subject text to a standard accounting subject library and generate structured financial data; The data storage module is used to store the standard accounting subject library, accounting formula rule library, mapping rules, and structured financial data and verification results.

6. The system according to claim 5, characterized in that, The accounting subject standardization module includes a synonym expansion set construction unit and a dual similarity calculation unit; the dual similarity calculation unit is used to calculate the literal similarity and semantic similarity between the accounting subject text and the standard accounting subject, and then weight the two to obtain a mapping score; When the mapping score is lower than the subject mapping threshold, the accounting subject text is marked as an item to be manually confirmed and pushed to the manual review interface.

7. The system according to claim 5, characterized in that, It also includes an accounting expert intelligent agent module, which is a rule-based expert system; when the accounting verification module reports that the accounting formula calculation has failed, the accounting expert intelligent agent module traces back the accounting subject data related to the failed formula based on the accounting rule knowledge base, locates one or more numerical cells that are suspected of being identified incorrectly, and generates repair suggestions for manual review.

8. The system according to claim 5, characterized in that, It also includes a Trusted Execution Environment (TEE) module and a formal constraint solver module; The Trusted Execution Environment (TEE) module is used to perform sensitive information region detection, desensitization, and seal removal processing of the preprocessing module within the Trusted Execution Environment; high-definition slice cropping of the original undesensitized image required for local rollback and re-identification of the hybrid OCR module; and local OCR inference. Before sending the seal-removed financial report image to the cloud, it generates a remote verification report bound to the metric value and program version of the Trusted Execution Environment. The cloud OCR interface in the hybrid OCR module only receives the seal-removed financial report image and returns the cloud recognition result when the remote verification report is verified successfully. The formal constraint solver module is used to formalize the accounting formula rule base in the accounting verification module into a constraint set, and construct a candidate value set for discrete variables by combining the cloud recognition results and the candidate values ​​output by the local OCR engine. An optimization model is established with the objective function of minimizing the accounting formula residual and minimizing the number of replaced cells and / or the replacement cost. The optimal solution that satisfies the constraint set is obtained by solving the model using Satisfiability Modulo Theories (SMT) or integer linear programming, so as to determine the final value of each target cell for outputting structured financial data.

9. The system according to claim 5, characterized in that, It also includes a digital forensics anti-forgery detection module, which is configured to perform tampering / forgery risk analysis on the financial report image before or after the preprocessing module outputs the image of the financial report with the seal removed, including at least one or more of the following detections: compression consistency detection based on error level analysis (ELA), compression parameter consistency detection based on JPEG quantization table / block effect, consistency detection based on image noise fingerprint (PRNU), and splicing anomaly detection based on layout geometric consistency and font / character spacing statistical features, and output an evidence risk score R_forensic; Specifically, when calculating the comprehensive score, the accounting verification module incorporates the evidence collection risk score R_forensic as a risk penalty item, so that the comprehensive score simultaneously reflects OCR confidence, image quality score, accounting formula residuals, and evidence collection risk. Furthermore, when the evidence collection risk score R_forensic reaches or exceeds a preset evidence collection threshold, at least one handling strategy is triggered: performing local rollback re-identification on the target cell related to the accounting formula residual, marking the corresponding structured financial data as pending manual review, and / or outputting tampering risk warning information, thereby further suppressing the risk of erroneous data entry caused by forged or tampered images on the basis of closed-loop verification of accounting residuals.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the financial report image data extraction and verification method according to any one of claims 1 to 4.

Images