Bank account statement intelligent identification optimization system and method based on multi-modal fusion

Abstract

The invention relates to the technical field of financial data processing, in particular to a bank reconciliation bill intelligent identification optimization system and method based on multi-modal fusion, and the method comprises the steps: obtaining an image, a text and selectable audio data of a reconciliation bill through a multi-modal data collection module; after preprocessing and feature extraction, geometric correction and character recognition are carried out by using an image correction and character analysis module; the method is characterized in that a pre-training large language model based on a Transform architecture is introduced as a core of a multi-modal fusion and structured output module, and the model deeply fuses visual features, text semantics and audio context information through a cross attention mechanism to realize deep understanding of the overall structure of a bill and transaction logic. And finally, directly outputting a standardized structured recognition result. According to the method, the accuracy and robustness of bank key information identification in a complex scene are remarkably improved, and the method can be directly applied to automatic processes such as financial auditing and financial risk control.

Description

Technical Field

[0001] This invention relates to the field of financial data processing technology, specifically to a smart recognition and optimization system and method for bank statements based on multimodal fusion. Background Technology

[0002] Bank statements serve as core documents recording financial transactions, and their automated recognition is crucial for improving the efficiency of financial operations. Currently, mainstream technologies rely on Optical Character Recognition (OCR), such as commercial or open-source tools like Tesseract and Baidu OCR. However, OCR technology is essentially a single-modal (image) processing technology, which has significant bottlenecks when dealing with complex real-world scenarios. (1) Strong dependence on format: For statement templates with different formats, a lot of customized training is required, and the generalization ability is poor; (2) Weak anti-interference ability: When faced with blurry, tilted, dirty, or stamped images, the recognition accuracy drops sharply; (3) Lack of semantic understanding: It can only recognize characters and cannot understand business logic such as "borrowing" direction and date format normalization, which is prone to errors.

[0003] In recent years, multimodal learning has shown potential in the field of document understanding. For example, although the LayoutLM series of models integrates text, layout and image information, they are mainly for general documents, do not introduce audio modalities, and are not deeply optimized for the transaction tables and amount logic (etc.) specific to bank statements, and cannot adapt to the high-precision recognition needs of financial scenarios.

[0004] Therefore, existing technologies have not yet been able to effectively solve the problem of high-precision and robust recognition of bank statements in complex scenarios.

[0005] Therefore, those skilled in the art have provided a smart recognition and optimization system and method for bank statements based on multimodal fusion to solve the problems mentioned in the background art. Summary of the Invention

[0006] To address the aforementioned technical issues, this invention provides a multimodal fusion-based intelligent recognition and optimization system and method for bank statements. By deeply integrating visual, textual, and speech features and combining the powerful reasoning capabilities of a large language model, it achieves accurate and robust recognition and structured output of key information on bank statements in complex scenarios.

[0007] The intelligent recognition and optimization system for bank statements based on multimodal fusion includes a multimodal data acquisition module, a data preprocessing module, a multimodal feature extraction module, an image correction and text parsing module, and a multimodal fusion and structured output module; the connection relationships between the modules are as follows: The output of the multimodal data acquisition module is connected to the input of the data preprocessing module; the output of the data preprocessing module is connected to the input of the multimodal feature extraction module and the input of the image correction and text parsing module; the output of the multimodal feature extraction module is connected to the first input of the multimodal fusion and structured output module; the output of the image correction and text parsing module is connected to the second input of the multimodal fusion and structured output module. The functional definitions of each module are as follows: The multimodal data acquisition module is used to collect multimodal data from bank statements. The multimodal data includes image data, text data, and optional associated audio data, wherein the associated audio data is used to assist in correcting text recognition ambiguities or supplementing transaction context information. The data preprocessing module is used to standardize various modal data, including denoising, enhancing and edge detection of image data, word segmentation and cleaning of text data, and denoising and resampling of audio data. The multimodal feature extraction module is used to extract high-level features from preprocessed data, including using a convolutional neural network to extract visual feature vectors of images, using a pre-trained language model to extract semantic feature vectors of text, and using an acoustic feature extraction algorithm to extract acoustic feature vectors of audio. The image correction and text parsing module is used to perform geometric correction on the preprocessed image and to identify and extract the text content in the image; it includes a target detection submodule based on a deep neural network, used to locate the key corner points of the statement image and calculate the correction parameters; and a text recognition submodule based on OCR technology, used to perform text detection and recognition on the corrected image. The multimodal fusion and structured output module is used to receive and fuse the visual feature vector, semantic feature vector, acoustic feature vector, and the identified text content. Its core is a pre-trained large language model based on the Transformer architecture. This model achieves deep interaction and alignment of cross-modal features through a multi-head cross-attention mechanism, comprehensively understands the global layout, semantic logic, and contextual association of the statement, and outputs a structured recognition result.

[0008] Preferably, the multimodal data acquisition module includes: The image acquisition unit supports acquiring statement images with a resolution of no less than 300 dpi via scanner or mobile terminal camera; The text extraction unit supports directly parsing text from electronic statements in PDF and Excel formats, or receiving preliminary text from OCR. The audio acquisition unit supports the acquisition of narration audio with a sampling rate of 16kHz related to account statements.

[0009] Preferably, the target detection submodule in the image correction and text parsing module adopts the YOLOv13 model finely tuned on a bank statement image dataset containing various rotation, tilt, and stain scenes; the text recognition submodule adopts the PaddleOCR toolkit.

[0010] Preferably, the pre-trained large language model in the multimodal fusion and structured output module is a Decoder-only Transformer model that has been further fine-tuned on financial text and labeled data; The model's input is a fused sequence consisting of visual feature vectors, text feature vectors, audio feature vectors, and task instruction prompts. This sequence is encoded and decoded through multiple layers of Transformer blocks within the model, ultimately generating structured data that conforms to a predetermined JSON Schema.

[0011] Preferably, in the multi-head attention layer of the large language model, a cross-modal attention sub-layer is set up to calculate the cross-attention weights between visual features, text features, and audio features, expressed by the formula: , in, Features derived from a mode and Features from another modality The dimension of the feature vector is used to achieve adaptive weight allocation among features of different modalities through this formula, thereby realizing information complementarity and enhancement between modalities.

[0012] The intelligent recognition optimization method for bank statements based on multimodal fusion includes the following steps: S1. Multimodal data acquisition: Acquire images, text, and optional audio data from the target bank's statement, where the audio data is used to help correct text recognition ambiguities or supplement transaction context information; S2. Data preprocessing: Perform standardized cleaning and enhancement operations on each modal data; S3. Image Correction and Text Parsing: Automatically perform geometric correction on the image and identify and extract the text content to obtain the original text data; S4. Multimodal feature extraction: Extract visual feature vectors from the corrected image, semantic feature vectors from the original text data, and acoustic feature vectors from the audio data; S5. Multimodal fusion and structured reasoning: The visual feature vector, semantic feature vector, acoustic feature vector, and original text data are input together into a pre-trained large language model based on the Transformer architecture. The model uses its internal cross-modal attention mechanism to align, weight, and deeply fuse information from different modalities, and comprehensively infers the complete structured information of the statement. S6. Output: Output a standardized structured data file containing account information and a list of transaction details.

[0013] Preferably, in step S5, the specific process of deep fusion includes: First, each feature vector is mapped to a unified feature space through a linear projection layer; This is then concatenated with the word-embedded text sequence and system instructions to form the model's input sequence; During the decoding process, the model uses a cross-attention mechanism to dynamically focus on and absorb the most relevant visual and audio feature information at each step of text generation.

[0014] Preferably, in step S6, the standardized structured data is in JSON format, and its structure is predefined as: {“Statement Number”:“value”,“Account Information”:{“Bank”:“value”,“Account Number”:“value”},“Transaction Details”:[{“Transaction Date”:“value”,“Summary”:“value”,“Counterparty”:“value”,“Income”:“value”,“Expenditure”:“value”,“Balance”:“value”},…]}, and the large language model ensures that the output strictly follows this format through fine-tuning based on format constraints.

[0015] The technical effects and advantages of this invention are as follows: 1. Significantly improved recognition accuracy and robustness: Through cross-validation and complementarity of multi-source information, even in cases of poor image quality or missing text, inference and completion can be performed using other modal information, resulting in an overall recognition accuracy that is more than 30% higher than that of traditional OCR.

[0016] 2. Possesses deep semantic understanding capabilities: The large language model can understand financial terms, date / amount formats, and transaction logic, automatically perform standardization and logical verification, and the output results can be directly used by downstream business systems.

[0017] 3. Strong system practicality and generalization: The modular design makes it easy to deploy and maintain; the method based on pre-trained large models only requires a small amount of domain data for fine-tuning to adapt to various bank statement formats, reducing the cost of customized development. Attached Figure Description

[0018] Figure 1 This is a structural diagram of the intelligent recognition and optimization system for bank statements based on multimodal fusion provided in the embodiments of this application; Figure 2 This is a flowchart of the intelligent recognition and optimization method for bank statements based on multimodal fusion provided in the embodiments of this application; Figure 3 This is a schematic diagram of the extended hammer handle in the intelligent recognition and optimization system and method for bank statements based on multimodal fusion provided in the embodiments of this application. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The embodiments of the present invention are given for illustrative and descriptive purposes only, and are not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described to better illustrate the principles and practical application of the invention, and to enable those skilled in the art to understand the invention and design various embodiments with various modifications suitable for a particular purpose.

[0020] Example 1

[0021] Please see Figures 1-3 This embodiment provides a bank statement intelligent recognition optimization system and method based on multimodal fusion, including... Example 1: System Deployment and Data Preparation 1. Hardware environment: A server equipped with an NVIDIA RTX A6000 GPU is used for model inference, a high-performance CPU server is used for data preprocessing, and acquisition devices such as a scanner with a resolution of no less than 300dpi and a microphone that supports a 16kHz sampling rate are used.

[0022] 2. Software environment: Ubuntu 20.04 LTS operating system, Python 3.9 programming language, PyTorch 2.0 deep learning framework, Transformers library - for large model calls and fine-tuning, PaddleOCR toolkit - for text recognition, Librosa library - for audio feature extraction.

[0023] 3. Training Dataset Construction: 50,000 bank statement images from 20 different banks were collected, covering various formats such as scans, photos, and screenshots, including complex scenarios such as rotation, tilt, stains, and stamp obstruction. These images were then finely annotated, including: the coordinates of the four corner points for correction, text regions and content for OCR training, and the final structured JSON information for fine-tuning the large model. In addition, approximately 5,000 voice explanations for abnormal transactions were simulated, covering typical scenarios such as refunds, failed transfers, and payroll processing.

[0024] Example 2: Detailed Explanation of the Identification Process Taking a slightly tilted corporate bank statement with a stamp obscured and accompanied by voice instructions as an example, the recognition process of this invention will be explained in detail: S1. Multimodal data acquisition: Acquire a 300dpi resolution statement image via a scanner - filename: statement.jpg, and simultaneously acquire the corresponding transaction explanation audio via a microphone - filename: explanation.wav, sampling rate: 16kHz.

[0025] S2. Data Preprocessing: Gaussian filtering is used to denoise the image data, histogram equalization is used to enhance the image, and Canny algorithm is used for edge detection. Spectral subtraction is used to denoise the audio data, and it is uniformly resampled to 16kHz. Since there is no electronic account statement text in this case, the text extraction unit is not working temporarily, and subsequent text data comes from the OCR recognition results.

[0026] S3. Image Correction and Text Parsing: Using a finely tuned YOLOv13 model on a bank statement dataset, the four corner points of the bank statement in the image are detected. The perspective transformation matrix is ​​calculated based on the corner point coordinates to perform geometric correction on the tilted bank statement image. Subsequently, PaddleOCR is used to perform text detection and recognition on the corrected image to obtain the original text list.

[0027] S4. Multimodal Feature Extraction: (1) Visual feature extraction: The corrected image is input into the finely tuned ResNet-50 model on the ImageNet dataset and the bank statement image dataset to extract the 2048-dimensional visual feature vector output by the pooling layer; (2) Text feature extraction: The original text obtained by OCR recognition is input into the BERT model after fine-tuning on the financial corpus, and the 768-dimensional vector corresponding to the [CLS] label is taken as the text semantic feature vector; (3) Audio feature extraction: The Librosa library is used to extract 40-dimensional MFCC acoustic feature vectors from the preprocessed audio data.

[0028] S5. Multimodal fusion and structured reasoning: (1) Input sequence construction: Construct a prompt word - "Prompt" - as the beginning of text input: "You are a professional bank statement analysis assistant. Please extract the following structured content based on the provided image features, text content, and supplementary voice information:" Account Information - Bank Name, Account Number; Transaction details - Transaction date, summary, counterparty, income, expenses, balance; And output strictly according to the specified JSON format. The prompt word is embedded, and the 2048-dimensional visual feature vector and 40-dimensional acoustic feature vector are mapped to 768-dimensional vector through a linear projection layer. The word embedding vector of the OCR text is then concatenated with the prompt word embedding vector in the order of "prompt word embedding, visual feature projection vector, acoustic feature projection vector, and text word embedding" to form a long sequence of input to the model.

[0029] (2) Cross-modal fusion inference: Input long input sequences into the LLaMA-7B model, which is finely tuned on financial text and labeled data - i.e., Decoder-only Transformer model; When the model decodes and generates each token in the JSON string, its internal cross-modal attention sublayer calculates the attention weights between features from different modalities: , in, Features derived from a mode and Features from another modality Let be the dimension of the feature vector, and This formula enables adaptive weight allocation among different modal features, thereby achieving information complementarity and enhancement between modalities, and dynamically focusing on and absorbing the most relevant cross-modal information.

[0030] For example, when generating the "summary" field, if the text recognized by OCR has ambiguity, the model can supplement and correct it using the semantic information of "payroll disbursement" in the audio features.

[0031] S6. Structured Result Output: The LLaMA-7B model decodes and generates structured data conforming to a predefined JSON Schema, as follows: { "Statement Number":"20240520001", Account Information:{"Bank of Account Opening":"Industrial and Commercial Bank of China, Beijing Chaoyang Branch", Account: "6222080200012345678" }, Transaction details:[ {"Transaction Date":"2024-05-20", Abstract: "Payroll Disbursement" Counterparty: "Beijing XX Technology Co., Ltd." "Revenue": 15000.00 "Expenditure": 0.00, Balance: 85620.30 } ] } This structured data can be directly returned to the caller via API, enabling it to connect with downstream financial auditing or financial risk control systems.

[0032] Example 3: Effect Verification To verify the technical effectiveness of this invention, a test set containing 10,000 bank statements was constructed, covering various scenarios such as clear, tilted, blurred, and stamp-occluded statements. The recognition performance of this invention's solution was compared with that of traditional commercial OCR solutions and a simple multimodal stitching baseline model. The test metric was the field-level F1-score, which is the core indicator combining accuracy and recall. The test results are as follows: Figure 3 As shown.

[0033] The test results show that the overall field-level F1 score of the proposed solution reaches 96.7%, which is more than 30% higher than the traditional commercial OCR solution and 11.3% higher than the simple multimodal splicing baseline model. The advantages are more significant, especially in fields such as "counterparties" and "summary" that rely on contextual semantic understanding, which fully demonstrates the effectiveness of the multimodal deep fusion architecture of the proposed solution.

[0034] Meanwhile, supplementary tests were conducted for scenarios where multimodal loss was not observed: when only image + text modality was input, the F1-score of the proposed solution was 93.2%; when only image + audio modality was input, the F1-score was 88.5%; even if a single modality had serious defects, the recognition effect could still be guaranteed by other modal information, verifying the robustness of the system.

[0035] in conclusion This invention effectively solves the problem of intelligent recognition of complex bank statements by using an innovative "image + text + audio" trimodal deep fusion architecture, combined with a pre-trained large language model finely tuned in the financial field and a cross-modal attention mechanism.

[0036] Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art and related fields based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described and explained in the present invention, unless otherwise specified or limited, shall be implemented according to conventional means in the art.

Claims

1. A bank statement intelligent recognition and optimization system based on multimodal fusion, characterized in that, include A multimodal data acquisition module is used to collect multimodal data from bank statements, including image data, text data, and optional associated audio data. The data preprocessing module is used to standardize various modal data, including denoising, enhancing and edge detection of image data, word segmentation and cleaning of text data, and denoising and resampling of audio data. The multimodal feature extraction module is used to extract high-level features from preprocessed data, including extracting visual feature vectors of images using convolutional neural networks, extracting semantic feature vectors of text using pre-trained language models, and extracting acoustic feature vectors of audio using acoustic feature extraction algorithms. The image correction and text parsing module is used to perform geometric correction on the preprocessed image and to identify and extract the text content in the image. It includes a target detection submodule based on deep neural networks, which is used to locate the key corner points of the statement image and calculate the correction parameters; and a text recognition submodule based on OCR technology, which is used to detect and recognize text in the corrected image. The multimodal fusion and structured output module is used to receive and fuse the visual feature vector, semantic feature vector, acoustic feature vector, and the identified text content. Its core is a pre-trained large language model based on the Transformer architecture. This model achieves deep interaction and alignment of cross-modal features through a multi-head cross-attention mechanism, comprehensively understands the global layout, semantic logic, and contextual association of the statement, and outputs a structured recognition result.

2. The intelligent recognition and optimization system for bank statements based on multimodal fusion according to claim 1, characterized in that, The output of the multimodal data acquisition module is connected to the input of the data preprocessing module; the output of the data preprocessing module is connected to the input of the multimodal feature extraction module and the input of the image correction and text parsing module; the output of the multimodal feature extraction module is connected to the first input of the multimodal fusion and structured output module; and the output of the image correction and text parsing module is connected to the second input of the multimodal fusion and structured output module.

3. The intelligent recognition and optimization system for bank statements based on multimodal fusion according to claim 1, characterized in that, The multimodal data acquisition module includes: The image acquisition unit supports acquiring statement images with a resolution of no less than 300 dpi via scanner or mobile terminal camera; The text extraction unit supports directly parsing text from electronic statements in PDF and Excel formats, or receiving preliminary text from OCR. The audio acquisition unit supports the acquisition of narration audio with a sampling rate of 16kHz related to account statements.

4. The intelligent recognition and optimization system for bank statements based on multimodal fusion according to claim 1, characterized in that, The object detection submodule in the image correction and text parsing module uses a YOLOv13 model finely tuned on a bank statement image dataset containing various rotation, tilt, and stain scenarios; the text recognition submodule uses the PaddleOCR toolkit.

5. The intelligent recognition and optimization system for bank statements based on multimodal fusion according to claim 1, characterized in that, The pre-trained large language model in the multimodal fusion and structured output module is a Decoder-only Transformer model that has been further fine-tuned on financial text and labeled data. The model's input is a fused sequence consisting of visual feature vectors, text feature vectors, audio feature vectors, and task instruction prompts. This sequence is encoded and decoded through multiple layers of Transformer blocks within the model, ultimately generating structured data that conforms to a predetermined JSON Schema.

6. The intelligent recognition and optimization system for bank statements based on multimodal fusion according to claim 5, characterized in that, In the multi-head attention layer of the large language model, a cross-modal attention sub-layer is set up to calculate the cross-attention weights between visual features, text features, and audio features, expressed by the formula: ， in, Features derived from a mode and Features from another modality The dimension of the feature vector is used to achieve adaptive weight allocation among features of different modalities through this formula, thereby realizing information complementarity and enhancement between modalities.

7. A method for intelligent recognition and optimization of bank statements based on multimodal fusion, characterized in that, The system applied to any one of claims 1 to 6 includes the following steps: S1. Multimodal Data Acquisition: Acquire image, text, and optional audio data from the target bank's statement; S2. Data preprocessing: Perform standardized cleaning and enhancement operations on each modal data; S3. Image Correction and Text Parsing: Automatically perform geometric correction on the image and identify and extract the text content to obtain the original text data; S4. Multimodal feature extraction: Extract visual feature vectors from the corrected image, semantic feature vectors from the original text data, and acoustic feature vectors from the audio data; S5. Multimodal fusion and structured reasoning: The visual feature vector, semantic feature vector, acoustic feature vector, and original text data are input together into a pre-trained large language model based on the Transformer architecture. The model uses its internal cross-modal attention mechanism to align, weight, and deeply fuse information from different modalities, and comprehensively infers the complete structured information of the statement. S6. Output: Output a standardized structured data file containing account information and a list of transaction details.

8. The intelligent recognition and optimization method for bank statements based on multimodal fusion according to claim 7, characterized in that, In step S5, the specific process of deep fusion includes: First, each feature vector is mapped to a unified feature space through a linear projection layer; This is then concatenated with the word-embedded text sequence and system instructions to form the model's input sequence; During the decoding process, the model uses a cross-attention mechanism to dynamically focus on and absorb the most relevant visual and audio feature information at each step of text generation.

9. The intelligent recognition and optimization method for bank statements based on multimodal fusion according to claim 7, characterized in that, In step S6, the standardized structured data is in JSON format, and its structure is predefined as: {"Statement Number":"value","Account Information":{"Bank":"value","Account Number":"value"},"Transaction Details":[{"Transaction Date":"value","Summary":"value","Counterpartner":"value","Income":"value","Expenditure":"value","Balance":"value"},...]}, and the large language model ensures that the output strictly follows this format through fine-tuning based on format constraints.

Images