Multimodal large model and image forensics: financial document pre-examination method and system

Abstract

This invention discloses a method and system for pre-auditing financial vouchers using a multimodal large-scale model and image forensics, relating to the fields of artificial intelligence and financial automation. The method includes: acquiring images of payment vouchers to be audited and performing multi-dimensional digital image forensics detection; inputting the detected images into a multimodal large-scale model, guiding the model through visual cues to perform semantic understanding and entity recognition, autonomously locating and extracting key fields, and mapping them to voucher extraction data output; acquiring voucher extraction data, reimbursement data, and official invoice data to construct a three-dimensional data set, performing dynamic tolerance comparison of amounts and consistency verification of customer names; summarizing the detection, extraction, and verification results, and generating a structured pre-audit report containing risk levels through preset risk assessment rules. This invention achieves end-to-end automation from image input to risk reporting, improving the efficiency and accuracy of financial voucher auditing.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and financial automation technology, specifically to a method and system for pre-screening financial documents using a multimodal large model and image forensics. Background Technology

[0002] With the advancement of enterprise digital transformation, the need for automation in financial reimbursement processes is becoming increasingly urgent. In traditional reimbursement review, finance personnel need to process a large number of payment vouchers, including payment software screenshots, bank transfer receipts, POS machine slips, and other types.

[0003] The existing technology has the following significant pain points: First, unstructured data is difficult to process and highly template-dependent. Traditional OCR (Optical Character Recognition) technology mainly relies on template matching with fixed coordinates or specific rules. However, payment vouchers are diverse, and the interface layouts of different payment platforms and application versions vary greatly. Whenever the payment software updates its interface or introduces a new voucher type, the OCR template needs to be redeveloped, resulting in high system maintenance costs and difficulty in compatibility with multiple types of payment vouchers.

[0004] Secondly, manual review is inefficient and prone to errors. The invoice information submitted by the person seeking reimbursement needs to be compared one by one with the information in the payment voucher images. Manual comparison is not only time-consuming and laborious, but also prone to visual fatigue when faced with a large number of documents, leading to misreporting or omissions.

[0005] Third, there is a lack of effective anti-tampering mechanisms. With the widespread use of image editing software, fraudulent activities such as fabricating payment vouchers and modifying payment amounts are becoming increasingly covert. Traditional verification systems only focus on text recognition and lack technical means to detect the authenticity of the images themselves. They cannot effectively identify whether images have been modified by image processing software or whether barcode information has been tampered with, posing significant risk control risks to enterprises.

[0006] In summary, there is an urgent need for an intelligent pre-screening system that can automatically adapt to various certificate formats through semantic understanding and has image anti-tampering detection capabilities. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention proposes a financial voucher pre-audit method and system based on multimodal large model and image forensics, aiming to solve the technical problems of difficulty in uniformly identifying heterogeneous payment vouchers, difficulty in detecting image tampering, and difficulty in automatically auditing multi-source data.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: On the one hand, a method for pre-examination of financial documents using multimodal large models and image forensics is provided, including the following steps: S1. Obtain the image of the payment voucher to be audited, and perform multidimensional digital image forensics detection. Multidimensional digital image forensics detection shall include at least one of the following detection methods: metadata analysis, barcode cross-validation, and error level analysis. S2. Input the payment voucher image that has been detected by multidimensional digital image evidence into the multimodal large model and perform heterogeneous voucher information extraction: guide the multimodal large model to perform semantic understanding and entity recognition on the payment voucher image through visual cues engineering, autonomously locate and extract key fields, and map the extracted key fields into voucher extraction data output in a preset format; S3. Perform multi-source data audit: acquire voucher extraction data extracted from payment voucher images, reimbursement data entered in the reimbursement system, and official invoice data obtained through the tax system interface, construct a three-dimensional data set, perform dynamic tolerance comparison on the amount in the three-dimensional data set, and perform consistency verification on the customer names in the three-dimensional data set. S4. Based on the detection results of S1, the extraction results of S2, and the comparison and verification results of S3, and through the preset risk assessment rules, generate a structured pre-screening report containing risk levels.

[0009] Furthermore, metadata analysis includes: Parse the EXIF ​​metadata and header information of the payment voucher image, and perform the following operations: Tag traversal and extraction: Traverse the directory of all EXIF ​​metadata, retrieve attribute tags containing key fields of software identification class, and extract the corresponding attribute description values; Feature matching: Convert the extracted attribute description values ​​to lowercase and perform a fuzzy match with a preset database of suspicious editing software signatures. If the match is successful, it is marked as having traces of editing software. Device consistency determination: Parse the file header information to identify the image format type, and parse the EXIF ​​metadata to extract the fingerprint of the shooting device. If there are traces of editing software, or the fingerprint of the shooting device is abnormally lost, it is marked as artificially generated risk or secondary processing risk.

[0010] Furthermore, barcode cross-validation includes: The barcode or QR code area in the payment voucher image is verified in parallel via dual channels. The parallel verification includes low-level decoding and visual recognition. By using the underlying decoding engine, the barcode or QR code is de-decoded to obtain the original encrypted metadata stored in the barcode or QR code as the first value. Through visual recognition, a multimodal large model is invoked to directly extract the visible plain text numbers corresponding to barcodes or QR codes from the image surface as the second value; The first value and the second value are compared digit by digit. If they do not match, it is determined that the surface value has been altered or the barcode is counterfeited, and it is marked as barcode information tampering.

[0011] Furthermore, the error level analysis includes: The payment voucher image is compressed twice using a preset quality factor. The pixel difference between the image before compression and the image after compression is calculated to obtain the error level distribution of the image. The error level distribution is converted into a luminance component and then into a floating-point matrix. The color space conversion difference of each 8×8 pixel block is independently statistically analyzed, and the error statistic of each pixel block is calculated. Based on the error statistics of all pixel blocks in the image, the 95th percentile value is normalized to generate a grayscale heatmap for review by auditors. The mean value of the high error region is calculated based on the grayscale heatmap and recorded as the error level analysis score. Based on error level analysis scoring, a comprehensive tampering score is calculated according to a preset weighting formula; The preset weight formula is:

[0012] in, To comprehensively manipulate the scoring; For error level analysis scoring, The block effect score is obtained by estimating the smoothness of the 8×8 pixel block boundary using an edge statistics algorithm. This is a penalty item for metadata anomalies. and Preset weighting coefficients; If the overall tampering score exceeds the preset threshold, it is marked as image tampering, and the grayscale heatmap is color-mapped to generate a semi-transparent red heatmap, which is then overlaid on the payment voucher image and output.

[0013] Furthermore, visual cueing engineering includes: The payment voucher image is assembled with preset prompts, which include role definition, task objectives and multiple logical constraints. The role is defined as setting the multimodal large model as a financial voucher review expert; The objective is to extract confidence information from payment voucher images, with the confidence information dynamically determined based on the voucher type. Multiple logical constraints include: The thought chain model constraint guides the multimodal large model to decompose the process of semantic understanding and entity recognition of payment voucher images, autonomously locating and extracting key fields into multiple sub-steps, and then reasoning step by step according to the sub-steps and outputting intermediate results. Multi-batch asynchronous verification constraint: The same payment voucher image is called multiple times for independent extraction using a multimodal large model. The results of the multiple extractions are cross-compared. If they are inconsistent, a secondary audit mark is triggered. Priority constraint for amount extraction: When there are multiple amount fields in the payment voucher image, the payment amount or the discounted price is extracted first. Line break concatenation logic constraint: Automatically concatenates text content spanning multiple lines into a complete sentence; Character loss prevention constraint: For character blurring caused by image quality, inference and completion are performed based on contextual semantics; Barcode number differentiation constraint: Differentiate the numbers below the barcode or QR code from the transaction amount.

[0014] Furthermore, in S2, when calling the multimodal large model through the streaming interface, the temperature parameter is set to below 0.2; the multimodal large model performs semantic understanding and entity recognition on the payment voucher image based on the visual cues engineering, gradually generating inference text fragments containing key fields, and asynchronously splicing the inference text fragments to obtain voucher extraction data.

[0015] Furthermore, dynamic tolerance comparison includes: assuming the amount in the voucher extraction data is... The amount in the reimbursement data is The amount in the official invoice data is When satisfied If the amount matches, it is determined that the amount does not match; otherwise, it is determined that the amount does not match, and calculation is performed. and The difference is provided for verification.

[0016] Furthermore, the consistency verification of customer names includes: Retrieve the name of the payee from the voucher extraction data and the name of the seller from the official invoice data. Perform exact string matching between the payee's name and the seller's name on the invoice; if a match is found, the payee is considered to be the same. If the match fails, the large model is invoked to perform semantic reasoning to determine whether there is an affiliated relationship, payment on behalf relationship or brand franchise relationship between the name of the payee and the name of the seller on the invoice. If such a relationship exists, the payee is determined to be the same. If semantic reasoning still fails to determine that the customer and merchant are consistent, the semantic model is invoked to retrieve the enterprise's business registration information through the network to determine whether there is a real-time parent-child relationship or related enterprise relationship between the name of the receiving customer and the name of the invoice seller. If such a relationship exists, the customer and merchant are determined to be consistent. If no match is found through exact matching, semantic reasoning, and web retrieval, the business entity is marked as questionable.

[0017] Furthermore, multi-source data auditing also includes duplicate expense detection: For multiple payment voucher images submitted by the same person within a preset time period, extract the amount, payee name, and order number information from the voucher extraction data corresponding to each payment voucher image; If multiple voucher data are extracted with the same amount difference within the preset range, the same name of the receiving merchant or a related relationship, and different order numbers, they will be marked as suspected duplicate reimbursement or merchant fraudulent invoicing risk.

[0018] On the other hand, a system for implementing the above-mentioned financial document pre-audit method is provided, comprising: The multidimensional digital image forensics module is used to acquire images of payment vouchers to be audited and perform multidimensional digital image forensics detection. The multidimensional digital image forensics detection includes at least one of the following detection methods: metadata analysis, barcode cross-validation, and error level analysis. The multimodal large model information extraction module is used to input the payment voucher image detected by the multidimensional digital image forensics module into the multimodal large model and perform heterogeneous voucher information extraction: the multimodal large model is guided by visual cues engineering to perform semantic understanding and entity recognition on the payment voucher image, autonomously locate and extract key fields, and map the extracted key fields into voucher extraction data output in a preset format; The multi-source data audit module is used to perform multi-source data audits, obtain voucher extraction data output from the multimodal large model information extraction module, reimbursement data entered into the reimbursement system, and official invoice data obtained through the tax system interface, construct a three-dimensional data set, perform dynamic tolerance comparison on the amount in the three-dimensional data set, and perform consistency verification on the customer names in the three-dimensional data set. The pre-screening report generation module is used to summarize the detection results, extraction results, comparison and verification results of the multi-dimensional digital image forensics module, the multi-modal large model information extraction module and the multi-source data audit module, and generate a structured pre-screening report containing risk levels through preset risk assessment rules.

[0019] Compared with the prior art, the present invention has the following beneficial effects: 1. Existing technologies mainly rely on single-dimensional image detection (such as detecting only EXIF ​​metadata or performing only OCR recognition), which makes it difficult to detect sophisticated image forgery and barcode tampering. This invention creatively combines error-level analysis, metadata analysis, and barcode cross-validation to achieve triple authentication from file attributes to pixel features to data consistency, effectively solving the technical pain point of traditional auditing in detecting sophisticated image forgery and generative counterfeiting.

[0020] 2. Existing OCR technology heavily relies on fixed coordinate templates. Whenever payment software updates its interface or introduces new voucher types, templates need to be redeveloped, resulting in high maintenance costs and difficulty in compatibility with multiple voucher types. This invention abandons the template-dependent approach of traditional OCR. By constructing a visual cueing project (setting a multimodal large model as a financial voucher review expert, combined with thought chain pattern constraints and multi-batch asynchronous verification constraints), it leverages the semantic understanding and reasoning capabilities of the multimodal large model to achieve adaptive parsing of different heterogeneous payment vouchers. Regardless of whether the voucher is a screenshot, photo, or scan, and regardless of layout changes, the system can accurately extract key entities without predefined templates, significantly improving the system's compatibility with new voucher types and greatly reducing system maintenance costs.

[0021] 3. In existing technologies, payment vouchers, expense reports, and invoices are reviewed independently, lacking an effective cross-validation mechanism. Furthermore, differences in the representation of customer names across different systems can easily lead to false alarms. This invention constructs a ternary verification model that includes voucher extraction data, expense report data, and official invoice data. For amount verification, a dynamic tolerance comparison mechanism is used for ternary cross-verification. For customer consistency verification, a three-layer progressive verification logic (exact matching, large-scale model semantic reasoning, and semantic model network retrieval) is employed. By obtaining enterprise registration information in real time to determine parent-child relationships, the false alarm rate caused by customer abbreviations or differences in entity names is effectively reduced. For duplicate reimbursement detection, multiple payment vouchers from the same claimant are analyzed to automatically identify suspected duplicate reimbursements, fraudulent invoicing by customers, and other risky behaviors. This invention achieves a fully automated closed loop from voucher identification to risk assessment, significantly improving audit efficiency and accuracy. Attached Figure Description

[0022] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] like Figure 1 As shown, the present invention provides a method for pre-examination of financial documents based on multimodal large model and image forensics, comprising the following steps: S1. Obtain the image of the payment voucher to be audited, and perform multidimensional digital image forensics detection. Multidimensional digital image forensics detection includes at least one of the following detection methods: metadata analysis, barcode cross-validation, and error level analysis. S2. Input the payment voucher image that has been detected by multidimensional digital image evidence into the multimodal large model and perform heterogeneous voucher information extraction: guide the multimodal large model to perform semantic understanding and entity recognition on the payment voucher image through visual cues engineering, autonomously locate and extract key fields, and map the extracted key fields into voucher extraction data output in a preset format; S3. Perform multi-source data audit: acquire voucher extraction data extracted from payment voucher images, reimbursement data entered in the reimbursement system, and official invoice data obtained through the tax system interface, construct a three-dimensional data set, perform dynamic tolerance comparison on the amount in the three-dimensional data set, and perform consistency verification on the customer names in the three-dimensional data set. S4. Based on the detection results of S1, the extraction results of S2, and the comparison and verification results of S3, and through the preset risk assessment rules, generate a structured pre-screening report containing risk levels. In this invention, S1 is responsible for anti-tampering detection at the image level, S2 for information extraction at the content level, S3 for cross-auditing at the data level, and S4 for comprehensive evaluation at the result level. These four steps form a fully automated closed loop of evidence collection → extraction → auditing → reporting, solving the technical problems of traditional auditing relying on manual labor, difficulty in compatibility with heterogeneous credentials, and inability to detect sophisticated tampering.

[0025] S1 receives the payment voucher image and performs multi-dimensional digital image forensics detection, which simultaneously performs three detection methods: metadata analysis, barcode cross-verification, and error-level analysis. Metadata analysis is used to detect whether the image has been processed by editing software and whether the device fingerprint is abnormal; barcode cross-verification is used to detect whether the barcode or QR code is consistent with the display on the voucher; error-level analysis is used to detect pixel-level tampering or splicing traces. If any detection method detects an anomaly, the system records the corresponding tampering mark (such as the presence of editing software traces, barcode information tampering, or image tampering).

[0026] In one specific implementation, metadata analysis includes: Parse the EXIF ​​(Exchangeable Image File Format) metadata and header information of the payment voucher image, and perform the following operations: Tag traversal and extraction: Traversing all EXIF ​​metadata Table of contents Retrieve attribute tags containing key fields of software identification class and extract the corresponding attribute description values; Feature matching: Convert the extracted attribute description values ​​to lowercase and perform a fuzzy match with a preset database of suspicious editing software signatures. If the match is successful, it is marked as having traces of editing software. Device consistency determination: Parse the file header information to identify the image format type, and parse the EXIF ​​metadata to extract the fingerprint of the shooting device. If there are traces of editing software, or the fingerprint of the shooting device is abnormally lost, it is marked as artificially generated risk or secondary processing risk.

[0027] The aforementioned payment voucher images include, but are not limited to, mobile payment screenshots, bank transfer receipts, POS machine sales slips, screenshots of successful payment pages from e-commerce platforms, payment notification screenshots, electronic receipts, and scanned / photographed copies of paper invoices or receipts. Image formats include JPEG, PNG, BMP, or images converted from PDF.

[0028] EXIF (Exchangeable Image File Format) is a set of metadata standards embedded in digital photos during the shooting process. By traversing all EXIF ​​metadata directories, the system retrieves attribute tags containing key fields identifying software, such as "Software," "Application," or "Creator," extracts the corresponding attribute description values, and performs a fuzzy match against a pre-defined database of suspicious editing software signatures. If a match is found, the system is marked as containing traces of editing software.

[0029] The header information of the payment voucher image is parsed to identify the image format type, and the EXIF ​​metadata is parsed to extract the fingerprint of the capturing device. The capturing device fingerprint refers to the information carried by the device when the image was taken, including but not limited to the device manufacturer, device model, shooting time, GPS latitude and longitude, and software identifier. This information is stored in the EXIF ​​metadata and is an important basis for determining the credibility of the image source.

[0030] When the header information is parsed and the image is identified as a JPEG, the JPEG standard contains EXIF ​​metadata. Further parsing the EXIF ​​metadata extracts information such as the device manufacturer, device model, shooting time, GPS latitude and longitude, and software identifiers, forming a complete fingerprint of the shooting device. If the above information is complete and there are no signs of editing software, the device fingerprint is considered normal.

[0031] When parsing the file header information reveals the image to be in PNG format, the PNG format standard does not contain EXIF ​​metadata, making it impossible to extract any fingerprint information from the capturing device. Therefore, it is determined that the capturing device fingerprint has been abnormally lost. Regardless of the image content, it is marked as artificially generated and therefore risky.

[0032] When the file header information is parsed and the image is identified as being in BMP or GIF format, these formats also do not contain EXIF ​​metadata. The system will also determine that the fingerprint of the shooting device is abnormally lost and mark it as artificially generated risk.

[0033] If there are traces of editing software, or if the fingerprint of the shooting device is abnormally lost (i.e., the fingerprint information of the shooting device cannot be extracted from the EXIF ​​metadata, such as PNG, BMP, GIF and other passive information formats), it will be marked as a risk of artificial generation or secondary processing.

[0034] Through the above metadata analysis, the credibility of the source of payment voucher images was automatically determined, effectively avoiding forgery by modifying key information such as amount and merchant name through image editing software and then resaving it. At the same time, it solved the technical problem that the source of passive format screenshots (such as PNG) cannot be traced because they cannot provide fingerprints of the shooting device. This provides a basic credibility determination basis for subsequent barcode cross-validation and error level analysis.

[0035] In one specific implementation, barcode cross-validation includes: The barcode or QR code area in the payment voucher image is verified in parallel via dual channels. The parallel verification includes low-level decoding and visual recognition. Through underlying decoding, the underlying decoding engine is called. The underlying decoding engine adopts the ZXing or ZBar open source library, which supports the de-decoding of one-dimensional barcode specifications such as EAN-13, Code 39, and Code 128, as well as two-dimensional barcode specifications such as QR Code, Data Matrix, and PDF417, and obtains the original encrypted metadata stored in the barcode or QR code as the first value. Through visual recognition, a multimodal large model is invoked to directly extract the visible plain text numbers corresponding to barcodes or QR codes from the image surface as the second value; The first value and the second value are compared digit by digit. If they do not match, it is determined that the surface value has been altered or the barcode is counterfeited, and it is marked as barcode information tampering.

[0036] The above comparison can accurately detect two types of tampering: altering the surface values ​​of the ticket without changing the code and altering the barcode without changing the ticket surface, thus making up for the shortcomings of a single detection method.

[0037] In one specific implementation, error level analysis includes: The payment voucher image is compressed twice with a preset quality factor (e.g., 90%). The pixel difference between the image before compression and the image after compression is calculated to obtain the error level distribution of the image, i.e., the difference map. The error level distribution is converted into a luminance component (L dimension) and then into a floating-point matrix to eliminate color interference and improve calculation accuracy; the color space conversion difference of each 8×8 pixel block is independently statistically analyzed, and the error statistics of each pixel block are calculated. Based on the error statistics of all pixel blocks in the image, the 95th percentile value is normalized to generate a grayscale heatmap, which accurately locates pixel-level splicing or modification traces for review by auditors. The mean value of the high error region is calculated based on the grayscale heatmap and recorded as the error level analysis score. Based on error level analysis scoring, a comprehensive tampering score is calculated according to a preset weighting formula; The preset weight formula is:

[0038] in, To comprehensively manipulate the scoring; The error level score reflects the intensity of modification at the pixel level; the higher the value, the higher the probability of pixel-level tampering. The block effect score is obtained by estimating the smoothness of the 8×8 pixel block boundary using an edge statistics algorithm. It is used to detect whether there is local resampling or splicing tampering. The more significant the block boundary smoothness abnormality, the higher the score. As a metadata anomaly penalty item, if a suspicious editing software signature (such as Photoshop, GIMP, etc.) or a missing shooting timestamp is detected, a preset risk weight will be automatically added. and The preset weighting coefficients are used to adjust the contribution ratio of error level analysis score and block effect score in the overall tampering score, respectively.

[0039] If the overall tampering score exceeds a preset threshold (e.g., 0.15), it is marked as image tampering. The grayscale heatmap is then color-mapped to generate a semi-transparent red heatmap, which is then overlaid on the payment voucher image for output, assisting financial auditors in visually identifying risks.

[0040] The aforementioned error level distribution can accurately pinpoint tampering and forgery traces at the pixel level. For example, JPEG compression uses discrete cosine transform in 8×8 pixel blocks, and each save introduces a specific quantization error into each block. Tampered areas (such as altered amounts or spliced ​​image elements) undergo different compression cycles than the original image, resulting in a significantly lower compression error level compared to the background. In the same image, if a region (such as the amount or date position) is modified or spliced ​​twice, the number of compression cycles it undergoes differs from the original image, leading to a significantly higher error level after recompression compared to the background region. Based on this, the system independently statistically analyzes each 8×8 pixel block, calculating the error statistics (such as mean and variance) for each block. By normalizing the differences to the 95th percentile, a grayscale heatmap is generated, where bright areas (bright / white) indicate areas with abnormal compression errors and suspected tampering. Furthermore, the mean of high-error areas is calculated based on the grayscale heatmap and recorded as the error level analysis score. This rating can quantitatively reflect the intensity of modification at the pixel level—the more severe the tampering, the denser the high-error areas. The higher the value. Finally, based on the error level analysis score, combined with the block effect score... and metadata anomaly penalty items The overall tampering score is calculated according to a preset weighting formula. It is used to determine whether an image has been tampered with.

[0041] In one specific implementation, the execution process of S2 is as follows: The payment voucher image, after passing through S1 multidimensional digital image forensics detection, is input into the multimodal large model. The multimodal large model can adopt the Qwen-VL series model, which has image understanding and text generation capabilities and can process image and text input simultaneously.

[0042] First, the system constructs a visual cueing project: it verifies the MimeType attribute of the uploaded file stream to ensure the data source is in image format, and uses the Media wrapper class to atomically assemble the payment voucher image binary resource with preset cue words. The cue words include role definitions, task objectives, and multiple logical constraints.

[0043] Role definition: The multimodal large model is set as a financial voucher auditing expert, enabling it to analyze image content from a professional financial auditing perspective.

[0044] Task objective: Extract confidence information from payment voucher images. This confidence information is dynamically determined based on the voucher type. For example, for payment screenshots, extract the merchant name, payment amount, transaction time, and order number; for bank transfer receipts, extract the payee name, transfer amount, transaction time, and transaction serial number.

[0045] Multiple logical constraints include: The thought chain model constraint guides the multimodal large model to decompose the process of semantic understanding and entity recognition of payment voucher images, autonomous localization and extraction of key fields into multiple sub-steps, and to reason step by step according to the sub-steps and output intermediate results, so that the reasoning process is interpretable and traceable.

[0046] Multi-batch asynchronous verification constraint: The multimodal large model is called multiple times to extract the same payment voucher image independently. The results of multiple extractions are cross-compared. If they are inconsistent, a secondary audit mark is triggered, and the extraction result with the highest confidence is retained, which effectively reduces the occasional errors of a single call.

[0047] Priority constraint for amount extraction: When there are multiple amount fields in the payment voucher image, the payment amount or the discounted price is extracted first to avoid mistakenly extracting the original price or deposit.

[0048] Line break concatenation logic constraint: Automatically concatenates text content across multiple lines into a complete sentence, solving the problem of broken lines in screenshot text.

[0049] Character loss prevention constraints: For characters blurred due to image quality issues (blur, reflection, occlusion), inference and completion are performed based on the contextual semantics.

[0050] Barcode number differentiation constraint: Differentiate the numbers below the barcode or QR code from the transaction amount numbers to avoid mistakenly extracting the order number as the amount.

[0051] Secondly, the large model parameters were configured, with the temperature parameter set to 0.2 to ensure output stability. The multimodal large model was then called via a streaming interface. Guided by the visual cues project, the multimodal large model performed semantic understanding and entity recognition on the payment voucher image. The model accurately located and extracted key fields such as payment amount, merchant name, and payment time through contextual semantics (such as text near "total," "payer," and "recipient"). Even when faced with a voucher format that had never been seen before, it could perform semantic alignment through logical constraints and autonomously identify and locate key entities.

[0052] During the extraction process, the model sequentially generates inference text fragments containing key fields, then receives these fragments in real time and performs asynchronous concatenation processing. For example, for a payment screenshot, the model might output text fragments sequentially; these fragments are received in real time and asynchronously concatenated, combining them into a complete JSON string in the order they are received to obtain the voucher extraction data. Finally, the recognition result is mapped to voucher extraction data in a preset format and output. The preset format can be a JSON schema, which includes at least one of the following: voucher type matching field, standardized merchant name field, amount field, transaction time field, and order number field.

[0053] In one specific implementation, the execution process of S3 is as follows: Perform multi-source data audit, obtain voucher extraction data (including merchant name, payment amount, transaction time, order number, etc.) output by S2, reimbursement data filled in by employees from the enterprise reimbursement system (including reimbursement amount, customer name, etc.), and official invoice data (including invoice amount, seller name, etc.) obtained through the tax system interface, and construct a three-data set.

[0054] In the dynamic tolerance comparison of amounts, the payment amount is extracted from the voucher data. The reimbursement amount in the reimbursement data Invoice amount in official invoice data Regarding the payment amount With invoice amount Minor discrepancies (such as handling fees, installment interest, or partial deductions) are addressed by introducing a dynamic tolerance calculation model: when the following conditions are met... If the amounts match, the system determines that the amounts do not match; otherwise, it determines that the amounts do not match and calculates the result. and The difference is provided for verification.

[0055] In the customer name consistency verification, the name of the receiving customer in the voucher extraction data and the name of the invoice seller in the official invoice data are obtained. First, an exact string match is performed. If the match fails, a large language model is called to perform semantic reasoning to determine whether there is a subordinate relationship, a payment on behalf relationship, or a brand franchise relationship between the two (for example, determining that "A" is a sub-brand of "B"). If they exist, the customer and merchant are determined to be consistent. If the semantic reasoning still fails to determine consistency, the semantic model is further called to retrieve the enterprise's business registration information through the network to determine the real-time parent-child relationship.

[0056] In addition, duplicate reimbursement detection is performed on multiple payment voucher images submitted by the same person within a preset time period (e.g., 30 days). If multiple voucher data are extracted with the amount difference within a preset range (the difference between the amounts of multiple payment voucher images does not exceed a preset threshold (e.g., ±5%)), the name of the receiving merchant is the same or there is a relationship, and the order number is not repeated, then it is marked as suspected duplicate reimbursement or merchant fraud risk.

[0057] Finally, the system summarizes the dynamic tolerance comparison results, consistency verification results, and duplicate reimbursement detection results, and outputs them as the verification results of S3 to S4.

[0058] In one specific implementation, the execution process of S4 is as follows: The system summarizes the outputs of S1, S2 and S3, and generates a structured pre-screening report containing risk levels according to preset risk assessment rules.

[0059] Specifically, S1 outputs multi-dimensional digital image forensic detection results, including whether there are traces of editing software, whether there is barcode information tampering, whether there is a risk of manual generation or secondary processing, whether there is barcode information tampering, whether there is image tampering, and the corresponding semi-transparent red heat map; S2 outputs voucher extraction data (JSON format, including merchant name, payment amount, transaction time, order number, etc.) extracted from the payment voucher image; S3 outputs multi-source data audit results, including whether the amount matches and the difference value, whether the customer and merchant entity is questionable, and whether there is a risk of suspected duplicate reimbursement or false invoicing by the customer.

[0060] Based on the preset risk assessment rules, the risk level is determined as follows: If S1 contains any tampering mark (editing software traces, barcode information tampering, image tampering) or S3 has a mismatch in amount with a difference exceeding 20%, it is considered high risk; if S1 has artificially generated risk or secondary processing risk, or S3 has a mismatch in amount with a difference between 10% and 20%, or S3 has a questionable customer entity but a semantic connection, or S2 has incomplete data extraction, it is considered medium risk; if none of the above anomalies exist and the extracted data is complete, it is considered low risk.

[0061] If S1 shows no signs of tampering (editing software traces, barcode information tampering, risks of manual generation or secondary processing, barcode information tampering, image tampering), S2 shows complete extracted data, and S3 shows matching amounts and consistent customer entities, the overall assessment is low risk.

[0062] Finally, the risk level, detailed results of each step, as well as the difference details, audit recommendations, and semi-transparent red heatmap (if any) are assembled into a structured pre-audit report in JSON format for quick review by finance personnel or automatic accounting processing.

[0063] In one specific implementation, the discrepancy details include amount discrepancy details and customer / vendor discrepancy details. The amount discrepancy details record the specific values ​​of the voucher withdrawal amount, reimbursement amount, and invoice amount, as well as the differences between each pair, and indicate the percentage difference. The customer / vendor discrepancy details record the comparison process between the name of the receiving customer and the name of the invoice seller (exact matching results, semantic reasoning results, and online search results) and the final judgment result. Financial personnel can quickly locate problems based on the discrepancy details without having to check the original documents item by item.

[0064] In one specific implementation, the review recommendations are automatically generated based on the risk level: for low risk, the recommendation is "Review passed, no manual review required"; for medium risk, the recommendation is "Manual review recommended" and a list of specific differences requiring close attention (such as...). and (For high-risk situations, such as discrepancies in the audit results or doubts about the customer's identity, etc.), it is recommended to "force manual review and refuse automatic approval" and specify the specific reasons for triggering the high risk (such as image tampering, barcode information tampering, etc.). Finance personnel should quickly make a decision to approve, reject, or require further verification based on the audit recommendations.

[0065] In one specific implementation, the present invention employs a multimodal large-scale model and a semantic large-scale model according to task requirements: the multimodal large-scale model is used for heterogeneous voucher information extraction in S2, assembling the payment voucher image with visual cues as input, and outputting JSON structured data containing key fields such as amount, merchant name, transaction time, and order number; the semantic large-scale model is used for merchant name semantic reasoning and web retrieval in S3, determining whether there is an affiliated relationship, payment on behalf relationship, or brand franchise relationship between the payee's name and the invoice seller's name, and can combine retrieval enhancement generation technology to retrieve enterprise business information through the web to determine real-time parent-child relationship or related enterprise relationship. The two models work together: the multimodal large-scale model extracts structured information from the image, and the semantic large-scale model performs semantic reasoning on the merchant name, jointly realizing the complete technical link from voucher extraction to merchant verification.

[0066] The present invention also provides a system for the above-mentioned financial document pre-approval method, comprising: The multidimensional digital image forensics module is used to acquire images of payment vouchers to be audited and perform multidimensional digital image forensics detection. The multidimensional digital image forensics detection includes at least one of the following detection methods: metadata analysis, barcode cross-validation, and error level analysis. The multimodal large model information extraction module is used to input the payment voucher image detected by the multidimensional digital image forensics module into the multimodal large model and perform heterogeneous voucher information extraction: the multimodal large model is guided by visual cues engineering to perform semantic understanding and entity recognition on the payment voucher image, autonomously locate and extract key fields, and map the extracted key fields into voucher extraction data output in a preset format; The multi-source data audit module is used to perform multi-source data audits, obtain voucher extraction data output from the multimodal large model information extraction module, reimbursement data entered into the reimbursement system, and official invoice data obtained through the tax system interface, construct a three-dimensional data set, perform dynamic tolerance comparison on the amount in the three-dimensional data set, and perform consistency verification on the customer names in the three-dimensional data set. The pre-screening report generation module is used to summarize the detection results, extraction results, comparison and verification results of the multi-dimensional digital image forensics module, the multi-modal large model information extraction module and the multi-source data audit module, and generate a structured pre-screening report containing risk levels through preset risk assessment rules.

[0067] Finally, it should be noted that the above embodiments are merely preferred embodiments of the present invention used to illustrate the technical solutions of the present invention, and are not intended to limit the invention, nor are they intended to limit the patent scope of the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention. That is to say, any changes or refinements made to the main design concept and spirit of the present invention that are not of substantial significance, but whose technical problems are still consistent with the present invention, should be included within the protection scope of the present invention. In addition, the direct or indirect application of the technical solutions of the present invention to other related technical fields are similarly included within the patent protection scope of the present invention.

Claims

1. A method for pre-examination of financial documents based on multimodal large model and image forensics, characterized in that, Includes the following steps: S1. Obtain the image of the payment voucher to be audited, and perform multidimensional digital image forensics detection. Multidimensional digital image forensics detection includes at least one of the following detection methods: metadata analysis, barcode cross-validation, and error level analysis. S2. Input the payment voucher image that has been detected by multidimensional digital image evidence into the multimodal large model and perform heterogeneous voucher information extraction: guide the multimodal large model to perform semantic understanding and entity recognition on the payment voucher image through visual cues engineering, autonomously locate and extract key fields, and map the extracted key fields into voucher extraction data output in a preset format; S3. Perform multi-source data audit: acquire voucher extraction data extracted from payment voucher images, reimbursement data entered in the reimbursement system, and official invoice data obtained through the tax system interface, construct a three-dimensional data set, perform dynamic tolerance comparison on the amount in the three-dimensional data set, and perform consistency verification on the customer names in the three-dimensional data set. S4. Based on the detection results of S1, the extraction results of S2, and the comparison and verification results of S3, and through the preset risk assessment rules, generate a structured pre-screening report containing risk levels.

2. The financial voucher pre-examination method according to claim 1, characterized in that, Metadata analysis includes: Parse the EXIF ​​metadata and header information of the payment voucher image, and perform the following operations: Tag traversal and extraction: Traverse the directory of all EXIF ​​metadata, retrieve attribute tags containing key fields of software identification class, and extract the corresponding attribute description values; Feature matching: Convert the extracted attribute description values ​​to lowercase and perform a fuzzy match with a preset database of suspicious editing software signatures. If the match is successful, it is marked as having traces of editing software. Device consistency determination: Parse the file header information to identify the image format type, and parse the EXIF ​​metadata to extract the fingerprint of the shooting device. If there are traces of editing software, or the fingerprint of the shooting device is abnormally lost, it is marked as artificially generated risk or secondary processing risk.

3. The financial voucher pre-examination method according to claim 1, characterized in that, Barcode cross-validation includes: The barcode or QR code area in the payment voucher image is verified in parallel via dual channels. The parallel verification includes low-level decoding and visual recognition. By using the underlying decoding engine, the barcode or QR code is de-decoded to obtain the original encrypted metadata stored in the barcode or QR code as the first value. Through visual recognition, a multimodal large model is invoked to directly extract the visible plain text numbers corresponding to barcodes or QR codes from the image surface as the second value; The first value and the second value are compared digit by digit. If they do not match, it is determined that the surface value has been altered or the barcode is counterfeited, and it is marked as barcode information tampering.

4. The financial voucher pre-examination method according to claim 1, characterized in that, Error level analysis includes: The payment voucher image is compressed twice using a preset quality factor. The pixel difference between the image before compression and the image after compression is calculated to obtain the error level distribution of the image. The error level distribution is converted into a luminance component and then into a floating-point matrix. The color space conversion difference of each 8×8 pixel block is independently statistically analyzed, and the error statistic of each pixel block is calculated. Based on the error statistics of all pixel blocks in the image, the 95th percentile value is normalized to generate a grayscale heatmap for review by auditors. The mean value of the high error region is calculated based on the grayscale heatmap and recorded as the error level analysis score. Based on error level analysis scoring, a comprehensive tampering score is calculated according to a preset weighting formula; The preset weight formula is: ； in, To comprehensively manipulate the scoring; For error level analysis scoring, The block effect score is obtained by estimating the smoothness of the 8×8 pixel block boundary using an edge statistics algorithm. This is a penalty item for metadata anomalies. and Preset weighting coefficients; If the overall tampering score exceeds the preset threshold, it is marked as image tampering, and the grayscale heatmap is color-mapped to generate a semi-transparent red heatmap, which is then overlaid on the payment voucher image and output.

5. The financial voucher pre-examination method according to claim 1, characterized in that, Visual cueing engineering includes: The payment voucher image is assembled with preset prompts, which include role definition, task objectives and multiple logical constraints. The role is defined as setting the multimodal large model as a financial voucher review expert; The objective is to extract confidence information from payment voucher images, with the confidence information dynamically determined based on the voucher type. Multiple logical constraints include: The thought chain model constraint guides the multimodal large model to decompose the process of semantic understanding and entity recognition of payment voucher images, autonomously locating and extracting key fields into multiple sub-steps, and then reasoning step by step according to the sub-steps and outputting intermediate results. Multi-batch asynchronous verification constraint: The same payment voucher image is called multiple times for independent extraction using a multimodal large model. The results of the multiple extractions are cross-compared. If they are inconsistent, a secondary audit mark is triggered. Priority constraint for amount extraction: When there are multiple amount fields in the payment voucher image, the payment amount or the discounted price is extracted first. Line break concatenation logic constraint: Automatically concatenates text content across multiple lines into a complete sentence; Character loss prevention constraint: For character blurring caused by image quality, inference and completion are performed based on contextual semantics; Barcode number differentiation constraint: Differentiate the numbers below the barcode or QR code from the transaction amount.

6. The financial voucher pre-examination method according to claim 1, characterized in that, In S2, when calling the multimodal large model through the streaming interface, the temperature parameter is set to below 0.

2. Based on the visual cues, the multimodal large model performs semantic understanding and entity recognition on the payment voucher image, gradually generating inference text fragments containing key fields. The inference text fragments are then asynchronously spliced ​​to obtain the voucher extraction data.

7. The financial voucher pre-examination method according to claim 1, characterized in that, Dynamic tolerance comparison includes: assuming the amount in the voucher extraction data is... The amount in the reimbursement data is The amount in the official invoice data is When satisfied If the amount matches, it is determined that the amount does not match; otherwise, it is determined that the amount does not match, and calculation is performed. and The difference is provided for verification.

8. The financial voucher pre-examination method according to claim 1, characterized in that, The consistency check for customer names includes: Retrieve the name of the payee from the voucher extraction data and the name of the invoice seller from the official invoice data; Perform exact string matching between the payee's name and the seller's name on the invoice; if a match is found, the payee is considered to be the same. If the match fails, the large model is invoked to perform semantic reasoning to determine whether there is an affiliated relationship, payment on behalf relationship or brand franchise relationship between the name of the payee and the name of the seller on the invoice. If such a relationship exists, the payee is determined to be the same. If semantic reasoning still fails to determine that the customer and merchant are consistent, the semantic model is invoked to retrieve the enterprise's business registration information through the network to determine whether there is a real-time parent-child relationship or related enterprise relationship between the name of the receiving customer and the name of the invoice seller. If such a relationship exists, the customer and merchant are determined to be consistent. If no match is found through exact matching, semantic reasoning, and web retrieval, the customer entity is marked as questionable.

9. The financial voucher pre-examination method according to claim 1, characterized in that, Multi-source data auditing also includes duplicate expense detection: For multiple payment voucher images submitted by the same person within a preset time period, extract the amount, payee name, and order number information from the voucher extraction data corresponding to each payment voucher image; If multiple voucher data are extracted with the same amount difference within the preset range, the same name of the receiving merchant or a related relationship, and different order numbers, they will be marked as suspected duplicate reimbursement or merchant fraudulent invoicing risk.

10. A system for implementing the financial document pre-examination method according to any one of claims 1-9, characterized in that, include: The multidimensional digital image forensics module is used to acquire images of payment vouchers to be audited and perform multidimensional digital image forensics detection. The multidimensional digital image forensics detection includes at least one of the following detection methods: metadata analysis, barcode cross-validation, and error level analysis. The multimodal large model information extraction module is used to input the payment voucher image detected by the multidimensional digital image forensics module into the multimodal large model and perform heterogeneous voucher information extraction: the multimodal large model is guided by visual cues engineering to perform semantic understanding and entity recognition on the payment voucher image, autonomously locate and extract key fields, and map the extracted key fields into voucher extraction data output in a preset format; The multi-source data audit module is used to perform multi-source data audits, obtain voucher extraction data output from the multimodal large model information extraction module, reimbursement data entered into the reimbursement system, and official invoice data obtained through the tax system interface, construct a three-dimensional data set, perform dynamic tolerance comparison on the amount in the three-dimensional data set, and perform consistency verification on the customer names in the three-dimensional data set. The pre-screening report generation module is used to summarize the detection results, extraction results, comparison and verification results of the multi-dimensional digital image forensics module, the multi-modal large model information extraction module and the multi-source data audit module, and generate a structured pre-screening report containing risk levels through preset risk assessment rules.

Images