A bank electronic receipt automatic recognition and RPA interface adaptation method and device based on a multi-modal large model

Abstract

A bank electronic receipt automatic recognition and RPA interface adaptation method and device based on a multi-modal large model, the main steps of which include: reading a bank electronic receipt PDF file, rendering the first page of the file into a high-resolution image; analyzing the first page image based on a multi-modal large model to obtain boundary coordinate data of each receipt; cutting the images of all pages of the PDF according to the boundary coordinates of the receipts obtained from the first page, separating each receipt from the original page, and generating independent receipt image files; extracting transaction information from each receipt image, and renaming the receipt image files according to the transaction identification number; organizing all the named receipt files into an ordered file archiving structure for subsequent batch calling of the RPA process. This method replaces the traditional scheme with the visual understanding capability of the multi-modal large model and can adapt to the receipt formats of various banks.

Description

Technical Field

[0001] This invention relates to the intersection of RPA and artificial intelligence, and in particular to a method and apparatus for automatic identification and RPA interface adaptation of bank electronic receipts based on a multimodal large model. Background Technology

[0002] In the daily operations of businesses in industries such as real estate and finance, electronic bank receipts are crucial documents for confirming fund receipts and completing the business loop. Companies typically need to match bank receipts with business orders in their systems to automate processes such as payment confirmation, financial reconciliation, and fund supervision. Robotic Process Automation (RPA) technology is widely used in these highly repetitive and rule-based business scenarios.

[0003] Currently, the mainstream automated bank statement processing solutions mainly adopt the following technical approaches: (1) Solution based on OCR + template rules: Extract text information from the receipt using OCR (Optical Character Recognition) technology, and then locate and extract key fields (such as transaction serial number, amount, date, etc.) by pre-setting fixed coordinate templates or regular expression rules for each bank. For example, China Construction Bank takes the first 19 digits after "Transaction Serial Number:" in the upper right corner, Bank of China takes the last 9 digits after "Transaction Serial Number:" in the lower right corner, Agricultural Bank of China takes the 4th to 13th digits of "Receipt Number:" in the upper left corner, and Bank of Beijing takes the second row of fields in the right frame of the transaction serial number in the lower right corner, etc.

[0004] (2) Semi-automated solution based on manual configuration: Business personnel manually write extraction rules for new banks, and developers implement them by coding and then deploy them into the RPA process.

[0005] However, the aforementioned prior art has the following significant drawbacks: First, the adaptation costs for different banks are extremely high. The receipt formats vary significantly between different banks, and even between different branches of the same bank, with substantial differences in the field names, positions, and formatting rules for transaction numbers / serial numbers. Currently, dozens of banks are known to require adaptation, including China Construction Bank, Bank of China, Bank of Beijing, Agricultural Bank of China, China Merchants Bank, Bank of Dalian, and Bank of Urumqi, each with its own unique receipt layout, field naming, and serial number extraction rules. Adding a new bank requires manual analysis of its receipt format, writing extraction rules, and developing adaptation code, resulting in a long development cycle and high maintenance costs.

[0006] Secondly, processing multiple receipts in a PDF file is difficult. Banks typically combine multiple receipts into a single PDF file with a fixed layout, and a single PDF may contain 100 or more receipts. Traditional PDF cutting solutions rely on hard-coded cutting using fixed page coordinates or pixel positions. Once the receipt layout changes (such as when the bank upgrades its receipt template), the cutting logic fails and needs to be redeveloped.

[0007] Third, the rules for extracting transaction serial numbers are fragile and inconsistent. Different banks use different names for transaction serial numbers ("transaction serial number," "receipt serial number," etc.), their positions are not fixed (top right, bottom right, top left, etc.), and their formats vary greatly (some include separators like "-" and "_", some require truncating a fixed number of characters, and some only require extracting the part before the separator). Extraction rules based on regular expressions or coordinates are highly susceptible to failure when faced with minor format adjustments, leading to RPA process interruptions.

[0008] Fourth, it lacks universality and scalability. The existing solution is essentially a customized development model of "one bank, one adaptation," which cannot be generalized to new bank statement formats that have not been seen before. When a company adds a new bank account or a partner bank upgrades its statement system, a complete analysis and development process needs to be carried out again, which seriously restricts the agility and scalability of the RPA process.

[0009] Therefore, designing a universal, adaptive automatic identification and data extraction method for different bank statement formats, so that it can be adapted to RPA processes through a unified interface, is a technical problem that urgently needs to be solved. Summary of the Invention

[0010] To address the aforementioned shortcomings of existing technologies, this disclosure provides a method and apparatus for automatic recognition and RPA interface adaptation of bank electronic receipts based on a multimodal large model. The core improvement lies in replacing the traditional OCR + template rule scheme with the visual understanding capability of the multimodal large model. By enabling the large model to directly "understand" the layout and content structure of the bank receipt, automatic receipt cutting and positioning, intelligent extraction of serial numbers, and bank type recognition are achieved. This allows for a unified processing flow that adapts to the receipt formats of all banks, eliminating the need to write separate extraction rules for each bank.

[0011] The method for automatic identification and RPA interface adaptation of bank electronic receipts based on a multimodal large model disclosed herein mainly includes the following steps: S1, obtain the bank's electronic receipt PDF file, which contains multiple receipts stitched together in a fixed layout, and render the first page of the PDF file as a high-resolution image; S2, use a multimodal large model to analyze the homepage image obtained in step S1, and obtain the boundary coordinate data of each order; S3, based on the obtained homepage order boundary coordinates, performs image segmentation on each page of the PDF, separates each order from the original page, and generates an independent order image file; S4. Based on the multimodal large model, transaction information is extracted from each receipt image, and the receipt image file is renamed with the transaction identifier. S5 archives and organizes all named return receipt files into an orderly file archive structure, which facilitates batch calling of subsequent RPA processes.

[0012] Furthermore, step S2 specifically includes: Input the homepage image obtained in step S1 into the multimodal large model, and guide the model to complete the following tasks using prompts: Identify the bank type to which the current PDF belongs; Analyze the arrangement of receipts on the page to determine the boundary area of ​​each receipt; The identified boundary information is returned in structured JSON format, and the output is a list of cropping coordinates for each order: Crops = {(x1,y1,w1,h1), (x2,y2,w2,h2), ...}.

[0013] Furthermore, the multimodal large model analysis described in step S2 includes the following elements in its prompt word design: Task Description: The model is explicitly required to identify bank types and transaction statement boundaries; Output format constraints: The model is required to return a list of bank types and clipping coordinates in JSON format; Contextual examples: Several known bank statement analysis examples are provided as few-shot references.

[0014] Furthermore, step S3 specifically includes: Based on the cropping coordinate list obtained through homepage analysis in step S2, the coordinates are applied to each page of the PDF to perform image cropping, separating each receipt from the original page and generating an independent receipt image file; For an N-page PDF, if each page contains M receipts, then N × M independent receipt image files will be generated.

[0015] Furthermore, step S4 specifically includes: For each receipt image generated in step S3, the multimodal large model is called again to extract key information and identify the transaction identifier in the receipt; The extracted transaction identifiers are cleaned according to preset standardization rules; Rename the receipt image file using the cleaned transaction identifier as the filename.

[0016] Furthermore, in step S4, the standardization rules for cleaning the extracted transaction identifiers specifically include: If the transaction identifier contains a separator "-" or "_", only the part before the separator is taken as a valid identifier; If the transaction identifier contains multiple delimiters, it is truncated at the position of the first occurrence of the delimiter. Remove leading and trailing whitespace characters from the extracted identifier; Ensure that the extracted result is a valid unique identifier string.

[0017] Furthermore, step S5 specifically includes: Create a folder named after the original PDF file name, and store all the receipt files named in step S4 in this folder to form an orderly file archive structure, which facilitates batch calling in subsequent RPA processes.

[0018] Furthermore, the method also includes the following steps: Based on the transaction identifier extracted from the multimodal large model in step S4, combined with the bank type information identified in step S2, standardized structured data is assembled, and a request is initiated through a unified RESTful API interface to push the receipt data to the business system to complete automatic matching. The request parameters for the API interface include: transaction identifier, bank type, and path to the receipt image file.

[0019] An automatic identification and RPA interface adaptation device for bank electronic receipts based on a multimodal large model, applying the above method, mainly includes: The file acquisition and preprocessing module is used to read bank electronic receipt PDF files and render the first page of the PDF file as a high-resolution image. The homepage analysis module is used to analyze the image of the first page of a PDF file and obtain the boundary coordinate data of each receipt on the first page. The PDF page cropping module is used to crop all pages of the PDF based on the order boundary coordinates obtained from the homepage, separating each order from the original page and generating an independent order image file. The transaction information extraction module is used to extract transaction information from each receipt image and rename the receipt image file with the transaction identifier number; The archive organization module is used to organize all named return receipt files into an ordered file archive structure, which facilitates batch calling in subsequent RPA processes.

[0020] A medium containing a computer program for performing the above-described method.

[0021] Compared with existing technologies, the beneficial effects of this disclosure are: ① High versatility: By replacing the traditional OCR + template rule solution with the visual understanding capability of the multimodal large model, there is no need to write extraction rules separately for each bank, and one process can be adapted to the receipt format of all banks; ② Adaptability: When a bank upgrades its receipt template or adds a new bank, the multimodal large model can automatically adapt to the new layout without manual re-analysis and development; ③ High processing efficiency: Only the first page of the PDF needs to be analyzed to derive the cutting rules for all pages, avoiding the overhead of page-by-page analysis; ④ Accurate serial number extraction: The multimodal large model can automatically locate and extract the transaction identifier based on semantic understanding, unaffected by changes in field position; ⑤ Low maintenance cost: Adapting to new banks requires only a few-shot examples or no additional configuration, significantly reducing development and maintenance costs. Attached Figure Description

[0022] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments of this disclosure taken in conjunction with the accompanying drawings, in which the same reference numerals generally represent the same components.

[0023] Figure 1 This is an overall flowchart of an exemplary embodiment according to the present disclosure; Figure 2 The flowchart for step S2, layout analysis based on a multimodal large model, is shown below. Figure 3 Here is a flowchart of step S4, serial number intelligent extraction and standardization processing. Figure 4 A detailed flowchart for step S6, API interface call. Detailed Implementation

[0024] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0025] This disclosure provides a method and apparatus for automatic identification and RPA interface adaptation of bank electronic receipts based on a multimodal large model.

[0026] In one exemplary implementation: The overall process is as follows: Figure 1 As shown, the specific implementation steps include: 1. Input and preprocessing of bank receipt PDF files.

[0027] Read bank electronic receipt PDF files from a specified directory. These PDF files are generated by the bank's system and contain multiple receipts arranged in a fixed layout on the same page. A single PDF file may contain more than 100 receipts, and the receipt formats, layouts, and field arrangements vary between different banks.

[0028] The first page of the PDF file is rendered as a high-resolution image and used as input for subsequent multimodal large model analysis. The reason for choosing the first page is that all pages within the same PDF file have the same layout, and the segmentation rules for all pages can be derived by analyzing only the first page.

[0029] 2. Order layout analysis and coordinate return based on multimodal large model.

[0030] Input the homepage image obtained in step 1 into the multimodal large model, and guide the model to complete the following tasks using carefully designed prompts: Identify the bank type to which the current PDF belongs (such as China Construction Bank, Bank of China, Agricultural Bank of China, etc.); Analyze the arrangement of receipts on the page to determine the boundary area of ​​each receipt (i.e., the start and end coordinates of each receipt on the page). The identified boundary information is returned in structured JSON format, and the output is a list of cropping coordinates for each order: Crops = {(x1,y1,w1,h1), (x2,y2,w2,h2), ...}.

[0031] The role of the multimodal large model in this step is to replace the traditional process of manually measuring coordinates or writing templates. Through the model's visual understanding capabilities, it directly perceives the arrangement pattern and boundary information of the slips from the image and returns the coordinate data in JSON format for the next step of cutting.

[0032] 3. PDF full-page cutting and image segmentation based on returned coordinates.

[0033] Obtain the cropping coordinate list returned by the multimodal large model in step 2. Apply these coordinates to each page of the PDF for image cropping, separating each receipt from the original page and generating an independent receipt image file (PNG format). Since step S2 has already obtained the receipt arrangement pattern through homepage analysis, this step applies this pattern to all pages of the PDF, performing batch cropping according to the same coordinate offset mode. Because the receipt layout is completely consistent across all pages within the same PDF file, only one coordinate analysis is needed to complete the cropping of all pages.

[0034] For an N-page PDF, if each page contains M receipts, then N × M independent receipt PNG image files will be generated.

[0035] 4. Intelligent extraction of serial numbers and file naming based on multimodal large models.

[0036] For each receipt PNG image generated in step 3, the multimodal large model is called again to extract key information: identify the transaction identifier number in the receipt (different banks call it "transaction serial number", "receipt number", etc.). The extracted transaction identifiers are cleaned according to preset normalization rules: 1) If the transaction identifier contains a separator "-" or "_", only the part before the separator is taken; 2) Remove leading and trailing whitespace characters; 3) Ensure that the extracted result is a valid unique identifier string.

[0037] Use the cleaned transaction identifier as the filename, and rename the receipt image to "transaction identifier.png".

[0038] The key improvement in this step is that the multimodal large model can automatically locate the position and field name of the transaction identifier based on the image content, without needing to know in advance where the bank will place the transaction number or what it will be called, thus achieving universal adaptation to different bank receipt formats.

[0039] 5. Archive and organize return receipt image files.

[0040] Create a folder named after the original PDF file name, and store all the receipt PNG files named in step 4 in this folder to form an orderly file archive structure, which facilitates batch calling in subsequent RPA processes.

[0041] 6. Structured data assembly and API interface calls.

[0042] Based on the transaction identifier extracted from the multimodal large model in step 4, and combined with the bank type information identified in step 2, standardized structured data is assembled. A request is then initiated through a unified RESTful API interface to push the receipt data to the business system for automatic matching. The API request parameters include fields such as transaction identifier, bank type, and receipt image file path.

[0043] The overall processing flow of this embodiment can be formally represented as the following pseudocode: Input: A collection of bank statement PDF files, D = {PDF1, PDF2, ..., PDF} n} Output: Transaction identifier and corresponding image file for each order, API matching results. For each PDF i ∈ D: (1) Transfer PDF i The homepage is rendered as the image IMG_first; (2) Call the multimodal large model analysis IMG_first to obtain: bank type bank_type, and the set of return receipt boundary coordinates Crops = {(x1,y1,w1,h1), (x2,y2,w2,h2), ...}; (3) For PDF i Each page j Apply the coordinates from Crops to the page. j The images are then cut and generated into individual return order images; (4) For each segmented receipt image (receipt_img): use the multimodal large model to extract the transaction identifier (txn_id); apply normalization rules to txn_id (remove the part after "-" and "_"); rename receipt_img to txn_id.png; (5) Create a folder named PDF i Store all txn_id.png files; (6) For each txn_id, assemble the API request parameters, call the business interface to complete the matching, and return the matching results of all return orders.

[0044] In this embodiment, the visual understanding capability of multimodal large models is applied to the analysis of bank receipt layout, realizing automatic cutting and positioning of receipts, replacing the traditional hard-coded coordinate scheme; A unified intelligent process for extracting transaction serial numbers was designed, which automatically adapts to the different naming and positions of transaction identifiers by different banks through the semantic understanding capabilities of multimodal large models. An efficient processing strategy of "homepage analysis + full-page application" is proposed, which can derive the segmentation rules of all pages by analyzing only the first page of the PDF; A standardized rule for transaction identifiers was designed to uniformly process serial numbers of different formats (including separators such as "-" and "_") from various banks, ensuring accurate matching. A unified RPA interface adaptation layer was built to enable unified access and automatic matching of different bank statement data through standardized API interfaces.

[0045] The following table compares specific bank receipts with traditional processing methods:

[0046] The above technical solutions are merely exemplary embodiments of the present invention. For those skilled in the art, based on the application methods and principles disclosed in the present invention, it is easy to make various types of improvements or modifications, and not limited to the methods described in the specific embodiments of the present invention. Therefore, the methods described above are merely preferred and not restrictive.

Claims

1. A method for automatic identification and RPA interface adaptation of bank electronic receipts based on a multimodal large model, characterized in that, Includes the following steps: S1, obtain the bank's electronic receipt PDF file, which contains multiple receipts stitched together in a fixed layout, and render the first page of the PDF file as a high-resolution image; S2, use a multimodal large model to analyze the homepage image obtained in step S1, and obtain the boundary coordinate data of each order; S3, based on the obtained homepage order boundary coordinates, performs image segmentation on each page of the PDF, separates each order from the original page, and generates an independent order image file; S4. Based on the multimodal large model, transaction information is extracted from each receipt image, and the receipt image file is renamed with the transaction identifier. S5 archives and organizes all named return receipt files into an orderly file archive structure, which facilitates batch calling of subsequent RPA processes.

2. The method according to claim 1, characterized in that, Step S2 specifically includes: Input the homepage image obtained in step S1 into the multimodal large model, and guide the model to complete the following tasks using prompts: Identify the bank type to which the current PDF belongs; Analyze the arrangement of receipts on the page to determine the boundary area of ​​each receipt; The identified boundary information is returned in structured JSON format, and the cropping coordinate list for each order is output: Crops = {(x1,y1,w1,h1), (x2,y2,w2,h2), ...}.

3. The method according to claim 2, characterized in that, The multimodal large model analysis described in step S2 includes the following elements in its prompt word design: Task Description: The model is explicitly required to identify bank types and transaction statement boundaries; Output format constraints: The model is required to return a list of bank types and clipping coordinates in JSON format; Contextual examples: Several known bank statement analysis examples are provided as few-shot references.

4. The method according to claim 1, characterized in that, Step S3 specifically includes: Based on the cropping coordinate list obtained through homepage analysis in step S2, the coordinates are applied to each page of the PDF to perform image cropping, separating each receipt from the original page and generating an independent receipt image file; For an N-page PDF, if each page contains M receipts, then N × M independent receipt image files will be generated.

5. The method according to claim 1, characterized in that, Step S4 specifically includes: For each receipt image generated in step S3, the multimodal large model is called again to extract key information and identify the transaction identifier in the receipt; The extracted transaction identifiers are cleaned according to preset standardization rules; Rename the receipt image file using the cleaned transaction identifier as the filename.

6. The method according to claim 5, characterized in that, In step S4, the normalization rules for cleaning the extracted transaction identifiers specifically include: If the transaction identifier contains a separator "-" or "_", only the part before the separator is taken as a valid identifier; If the transaction identifier contains multiple delimiters, it is truncated at the position of the first occurrence of the delimiter. Remove leading and trailing whitespace characters from the extracted identifier; Ensure that the extracted result is a valid unique identifier string.

7. The method according to claim 1, characterized in that, Step S5 specifically includes: Create a folder named after the original PDF file name, and store all the receipt files named in step S4 in this folder to form an orderly file archive structure, which facilitates batch calling in subsequent RPA processes.

8. The method according to any one of claims 1-7, further comprising the step of: Based on the transaction identifier extracted from the multimodal large model in step S4, combined with the bank type information identified in step S2, standardized structured data is assembled, and a request is initiated through a unified RESTful API interface to push the receipt data to the business system to complete automatic matching. The request parameters for the API interface include: transaction identifier, bank type, and path to the receipt image file.

9. A device for automatic identification and RPA interface adaptation of bank electronic receipts based on a multimodal large model, characterized in that, include: The file acquisition and preprocessing module is used to read bank electronic receipt PDF files and render the first page of the PDF file as a high-resolution image. The homepage analysis module is used to analyze the image of the first page of a PDF file and obtain the boundary coordinate data of each receipt on the first page. The PDF page cropping module is used to crop all pages of the PDF based on the order boundary coordinates obtained from the homepage, separating each order from the original page and generating an independent order image file. The transaction information extraction module is used to extract transaction information from each receipt image and rename the receipt image file with the transaction identifier number; The archive organization module is used to organize all named return receipt files into an ordered file archive structure, which facilitates batch calling in subsequent RPA processes.

10. A medium storing a computer program for performing the method of any one of claims 1-8.

Images