Financial certificate image content extraction method and device based on large model

Abstract

The invention discloses a financial certificate image content extraction method and device based on a large model, relates to the technical field of artificial intelligence and can also be used in the financial field, and the method comprises the steps: carrying out the element extraction and structured processing of a financial certificate image, and generating structured financial certificate information; performing semantic extraction on prompt words input by a user based on the structured financial certificate information through the large model to generate initial extraction content; based on the structured financial certificate information, performing verification and restoration processing on the initial extraction content to generate target extraction content, and accurately analyzing the complex format and table logic in the financial certificate image through deep structured processing to ensure the integrity of information extraction; on this basis, semantic extraction is performed by using a large model, and through an automatic verification and repair mechanism based on structured information, high accuracy and format consistency of an output result are effectively guaranteed, and the automation level, adaptability and reliability of processing various financial certificates are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and can also be used in the financial field, particularly to a method and apparatus for extracting the content of financial document images based on large models. Background Technology

[0002] With the rapid development of financial services, financial certificates, as important carriers of transaction information, are becoming increasingly diverse in type and format, making the demand for automated extraction of certificate information more urgent. Optical Character Recognition (OCR) technology, as a fundamental means of extracting text information from images, has been widely used in this field. However, traditional extraction methods based on rules or customized models generally suffer from insufficient generalization ability when faced with complex and diverse financial certificates. For each new certificate type or format change, a large amount of repetitive annotation, training, and optimization work is usually required, resulting in high model development and maintenance costs, low efficiency, and difficulty in adapting to flexible and ever-changing business needs.

[0003] In recent years, with breakthroughs in Large Scale Language Modeling (LLM) technology, its powerful semantic understanding and few-shot generalization capabilities have provided new approaches to information extraction tasks. Existing technologies include methods combining OCR with large models, such as identifying the information domain of text and constructing corresponding prompts to guide large models in structured information extraction. These methods reduce reliance on massive amounts of labeled data to some extent, improve the model's adaptability to different scenarios, and thus help reduce development costs and accelerate deployment. However, these existing solutions still have significant limitations when applied to financial document processing. First, financial documents are often presented in complex tabular form, and existing technologies typically lack effective parsing capabilities for table structures, especially struggling to accurately handle complex layouts such as row and column spans. This causes the text sequence output by OCR to lose its original logical connection, resulting in the omission or misalignment of key information, severely affecting the completeness of the extracted content. Second, the inherent randomness and uncontrollability of large model-generated results pose a challenge in scenarios like finance, where data accuracy and format consistency are extremely important. The model may produce fictitious content or output results with unstable formats, which directly affects the reliability and direct usability of the extracted results. This often requires additional complex post-processing steps for correction, increasing the uncertainty and complexity of the system.

[0004] In summary, current automated financial document information extraction technologies still face significant challenges in accurately parsing the complex format structures of documents (especially tables) and ensuring the accuracy and stability of large-scale model outputs. This limits the effectiveness and breadth of application of these technologies in real-world financial business scenarios, necessitating a more robust, accurate, and efficient solution.

[0005] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0006] This invention provides a method for extracting content from financial document images based on a large model. The method includes:

[0007] The acquired financial document images are subjected to element extraction and structuring processing to generate structured financial document information;

[0008] Using a large model, semantic extraction is performed on the prompts input by the user based on structured financial certificate information to generate initial extracted content;

[0009] Based on structured financial certificate information, the initial extracted content is verified and repaired to generate the target extracted content.

[0010] Preferably, the acquired financial document image is subjected to element extraction and structuring processing to generate structured financial document information, including:

[0011] The orientation of the financial document image is corrected to generate a corrected financial document image.

[0012] The pre-trained layout analysis model is used to extract elements from the corrected financial certificate image and generate certificate element information.

[0013] The information of the voucher elements is processed in a structured manner to generate structured financial voucher information.

[0014] Preferably, the financial document image is subjected to orientation correction processing to generate a corrected financial document image, including:

[0015] Text detection algorithms are used to detect text lines in financial document images and generate candidate text line regions.

[0016] A lightweight classification model is used to identify the direction of candidate text line regions and generate multiple candidate directions.

[0017] The overall direction of the text is determined based on multiple candidate directions;

[0018] Based on the overall orientation of the text, the orientation of the financial certificate image is corrected to generate a corrected financial certificate image.

[0019] Preferably, the method further includes:

[0020] Based on a pre-acquired document layout dataset, the object detection algorithm is trained to build a layout analysis model.

[0021] Preferably, a pre-trained layout analysis model is used to extract elements from the corrected financial document image to generate document element information, including:

[0022] Multi-scale feature detection is performed on the corrected financial certificate image to generate candidate layout blocks and corresponding element coordinates;

[0023] Classify the feature vectors of candidate page blocks to generate element classifications;

[0024] Content recognition is performed on different element categories to obtain the corresponding element content;

[0025] Post-processing is performed based on element coordinates, element category, and element content to generate voucher element information.

[0026] Preferably, based on structured financial certificate information, the initial extracted content is verified and repaired to generate the target extracted content, including:

[0027] Perform syntax validation on the initial extracted content;

[0028] If the syntax validation passes, the initial extracted content is mapped and validated using a pre-defined rule base and regular expressions based on the structured financial certificate information.

[0029] If the mapping and regular expression validation pass, the initial extracted content will be determined as the target extracted content;

[0030] If the mapping or regular expression validation fails, the initial extracted content is repaired based on the structured financial certificate information to generate the target extracted content;

[0031] If syntax validation fails, repeat the process of using a large model to semantically extract the prompts from the user's input based on structured financial certificate information to generate the initial extracted content.

[0032] Preferably, the method further includes:

[0033] The extracted target content is post-processed using regular expressions to generate the processed extracted target content.

[0034] This invention also provides a financial certificate image content extraction device based on a large model, the device comprising:

[0035] The element structuring processing unit is used to extract elements and perform structuring processing on the acquired financial certificate images to generate structured financial certificate information.

[0036] The semantic extraction unit is used to extract semantics from prompts input by users based on structured financial certificate information using a large model, and generate initial extracted content.

[0037] The content extraction unit is used to verify and repair the initial extracted content based on structured financial certificate information, and generate the target extracted content.

[0038] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for extracting financial document image content based on a large model.

[0039] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for extracting financial document image content based on a large model.

[0040] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for extracting financial document image content based on a large model.

[0041] The financial document image content extraction method and apparatus based on a large model provided in this invention extracts elements and performs structuring processing on the acquired financial document image to generate structured financial document information. Using a large model, semantic extraction is performed on user-input prompts based on the structured financial document information to generate initial extracted content. Based on the structured financial document information, the initial extracted content is verified and repaired to generate target extracted content. Through deep structuring processing, complex layouts and table logic in the financial document image are accurately analyzed, ensuring the completeness of information extraction. Furthermore, semantic extraction is performed using a large model, and an automated verification and repair mechanism based on structured information effectively ensures high accuracy and format consistency of the output results, significantly improving the automation level, adaptability, and reliability of processing various financial documents. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0043] Figure 1 A flowchart illustrating a method for extracting content from financial document images based on a large model, provided in an embodiment of the present invention;

[0044] Figure 2 A flowchart of another method for extracting financial document image content based on a large model, provided for an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of the structure of a financial certificate image content extraction device based on a large model provided in an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of the physical structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0048] The information collected in the technical solution of this application is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant countries and regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation portals are provided for users to choose to authorize or refuse.

[0049] The acquisition, transmission, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0050] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0051] To facilitate understanding of the technical solution provided in this application, the relevant content of the technical solution is explained below. This invention proposes a standardized OCR processing format that integrates page layout information. By enhancing the parsing ability of page layouts and table structures, it significantly improves the accuracy of information location, especially suitable for extracting financial voucher information in table form. Its main technical points include: structured page layout recognition, semantic extraction based on a large model, and a result verification and repair mechanism. Specifically, structured page layout recognition involves designing a unified page layout recognition standard to achieve effective parsing of complex page layouts, especially table structures; semantic extraction based on a large model utilizes the powerful generalization and semantic understanding capabilities of the large model to adapt to diverse voucher types and reduce customized development costs; the result verification and repair mechanism performs key-value pair verification and logical repair on the model output through post-processing, effectively suppressing the randomness and illusion problems of generated content and improving the accuracy and stability of the output results.

[0052] The following uses a large-model-based financial document image content extraction device as an example to illustrate the implementation process of the large-model-based financial document image content extraction method provided in this embodiment of the invention. It is understood that the execution entity of the large-model-based financial document image content extraction method provided in this embodiment of the invention includes, but is not limited to, a large-model-based financial document image content extraction device.

[0053] Figure 1 A flowchart of a method for extracting content from financial document images based on a large model, as provided in this embodiment of the invention, is shown below. Figure 1 As shown, the method includes:

[0054] Step 101: Extract elements and perform structuring processing on the acquired financial certificate image to generate structured financial certificate information.

[0055] In this embodiment of the invention, elements are extracted from the acquired financial document image, and different elements are identified, such as tables, titles, paragraphs, etc. It can not only extract text content, but also identify and retain spatial and logical information such as layout and table structure. By utilizing the effective area of ​​the layout, background and other interfering information are excluded. The identified elements are converted into well-structured data (JSON format), laying the foundation for subsequent processing.

[0056] Step 102: Using a large model, semantic extraction is performed on the prompt words input by the user based on the structured financial certificate information to generate initial extracted content.

[0057] In this embodiment of the invention, structured financial certificate information is combined with a large model, and user-defined prompts guide the large model to understand and extract the required business elements, forming preliminary extracted content.

[0058] Step 103: Based on the structured financial certificate information, perform verification and repair processing on the initial extracted content to generate the target extracted content.

[0059] In this embodiment of the invention, in order to address the problems of inconsistent formats and content deviations that may exist in the output of large models, the structured financial certificate information is used as the benchmark truth value to automatically verify, correct and standardize the format of the initial extracted content, thereby outputting highly accurate and highly consistent final target content.

[0060] In the technical solution provided by this invention, the acquired financial certificate image is subjected to element extraction and structuring processing to generate structured financial certificate information. Using a large model, semantic extraction is performed on the user-input prompts based on the structured financial certificate information to generate initial extracted content. Based on the structured financial certificate information, the initial extracted content is verified and repaired to generate target extracted content. Through deep structuring processing, the complex layout and table logic in the financial certificate image are accurately analyzed, ensuring the completeness of information extraction. Furthermore, semantic extraction is performed using a large model, and an automated verification and repair mechanism based on structured information effectively ensures the high accuracy and format consistency of the output results, significantly improving the automation level, adaptability, and reliability of processing various financial certificates.

[0061] Figure 2 A flowchart of another method for extracting financial document image content based on a large model, as provided in this embodiment of the invention, is shown below. Figure 2 As shown, the method includes:

[0062] Step 201: Perform orientation correction processing on the financial document image to generate a corrected financial document image.

[0063] In this embodiment of the invention, each step is performed by a financial certificate image content extraction device based on a large model.

[0064] In this embodiment of the invention, since the scanning or photographing of financial documents may result in tilting, which directly affects the accuracy of subsequent layout analysis and text recognition, it is necessary to perform orientation correction processing on the financial document images.

[0065] In this embodiment of the invention, step 201 specifically includes:

[0066] Step 2011: Use a text detection algorithm to detect text lines in the financial document image and generate candidate text line regions.

[0067] In this embodiment of the invention, a text detection algorithm is used to process the input financial certificate image, quickly and accurately locating the regions where all text lines are located in the financial certificate image, and generating multiple candidate text line regions.

[0068] In this embodiment of the invention, the text detection algorithm includes an efficient and accurate scene text detector (EAST) or a differentiable binarization network (DBNet).

[0069] Furthermore, the corrected financial document images undergo image preprocessing, including unifying image resolution, correcting skew / perspective distortion, and denoising filtering, to highlight foreground content and thus ensure the accuracy of subsequent recognition.

[0070] Step 2012: Using a lightweight classification model, perform orientation recognition on the candidate text line regions to generate multiple candidate orientations.

[0071] In this embodiment of the invention, each candidate text line region is cropped out, input into a lightweight classification model for direction determination, and the candidate direction corresponding to each candidate text line region is output.

[0072] It is worth noting that the lightweight classification model is the MobileNet model, which is trained to output the direction category of text lines, such as: 0° (positive), 90° (clockwise rotation of 90 degrees), 180°, 270°, etc.

[0073] Step 2013: Determine the overall direction of the text based on multiple candidate directions.

[0074] In this embodiment of the invention, candidate directions for all text line regions are counted, and the majority rule is adopted to determine the candidate direction with the largest number of candidates as the overall text direction, that is, the dominant overall direction of the text in the entire financial certificate image.

[0075] Step 2014: Based on the overall text orientation, perform orientation correction on the financial certificate image to generate the corrected financial certificate image.

[0076] In this embodiment of the invention, the financial certificate image is subjected to corresponding rotational geometric transformation according to the overall orientation of the text to generate a corrected financial certificate image, ensuring that the text in the image is in a standard horizontal orientation, providing positive input for subsequent analysis.

[0077] For example, if the overall orientation of the text is 90° (90 degrees clockwise), then the financial certificate image will be adjusted by -90° (90 degrees counterclockwise) to generate a corrected financial certificate image.

[0078] This invention ensures that financial document images are in a standard upright state before subsequent processing by detecting and correcting text line orientation. This provides a high-quality input foundation for layout structure analysis and character recognition, effectively avoids element missegmentation and coordinate misalignment caused by image tilt, and improves the robustness of the overall processing flow.

[0079] Step 202: Extract elements from the corrected financial document image using a pre-trained layout analysis model to generate document element information.

[0080] In this embodiment of the invention, the layout analysis model is constructed as follows: based on a pre-acquired document layout dataset, the object detection algorithm is trained to construct the layout analysis model. The document layout dataset is a publicly available dataset, such as PubLayNet and Marmot; the object detection algorithm includes Faster R-CNN or the YOLO series of algorithms (YOLOv8); and the model training is specifically supervised training, enabling the layout analysis model to identify different element types in financial document images.

[0081] In this embodiment of the invention, step 202 specifically includes:

[0082] Step 2021: Perform multi-scale feature detection on the corrected financial certificate image to generate candidate layout blocks and corresponding element coordinates.

[0083] In this embodiment of the invention, the corrected financial certificate image is input into the layout analysis model. Anchor points are generated by adapting the document region aspect ratio through its backbone network, multi-scale features are extracted, and a series of candidate layout blocks containing different layout elements are generated using the non-maximum suppression (NMS) algorithm. The feature vector of each candidate layout block and the precise coordinates of its effective content are output, usually in the format of (x1, y1, x2, y2) to represent the coordinates of the upper left and lower right corners.

[0084] Step 2022: Classify the feature vectors of candidate page blocks to generate element classifications.

[0085] In this embodiment of the invention, the feature vector extracted from each candidate page block is judged by the classification branch of the page analysis model to generate element classifications. Element classifications include, but are not limited to: titles, paragraphs, tables, headers, footers, images, formulas, etc.

[0086] Step 2023: Perform content recognition on different element categories to obtain the corresponding element content.

[0087] In this embodiment of the invention, specific content recognition is performed for different element regions after classification. Specifically, for text elements (titles, paragraphs, etc.), an OCR engine is used to recognize their text content; for table elements, not only the text within cells is recognized, but also the row and column structure is identified and converted into standard structured formats such as Markdown; for formula elements, a dedicated formula recognition engine is used to convert them into LaTeX format; for image elements, descriptive text can be generated by combining a visual language model (VLM), or chart-type images can be analyzed in depth and structured.

[0088] Step 2024: Perform post-processing based on element coordinates, element classification, and element content to generate voucher element information.

[0089] In this embodiment of the invention, post-processing includes, but is not limited to: merging overlapping or adjacent similar regions, correcting possible classification errors based on context and positional relationships (e.g., correcting the top region to a header), filtering out excessively small noise regions, and finally integrating element coordinates, element classification, and element content to generate well-organized voucher element information.

[0090] Step 203: Perform structuring processing on the voucher element information to generate structured financial voucher information.

[0091] This invention uses a deep learning model to accurately detect, classify, and identify the content of multi-level elements on a voucher layout. It not only extracts text but, more importantly, reconstructs the logical structure and spatial relationships of complex layouts, including tables, generating semantically rich structured information. This provides a complete, accurate, and easy-to-understand data foundation for subsequent semantic extraction, fundamentally solving the problems of missing and fragmented information in complex layouts.

[0092] In this embodiment of the invention, discrete voucher element information is organized into a unified and standardized structured financial voucher information according to its logical relationship and layout. It is typically encapsulated in a machine-readable format such as JSON. For example, an example of structured information containing a title, table, and paragraphs is as follows: [

[0094] {"Title": "xxx"},

[0095] {"Paragraph": "xxx"},

[0096] {"Table": {"Table Name": "xxx", "Table Content": "Table Markdown Format"}},

[0097] {"Formula": "Formula in LaTeX format"},

[0098] {"Header": "xxx"},

[0099] {"footer": "xxx"},

[0100] {"Image": {"Image Name": "xxx", "Image Content": "xxx"}} ]

[0102] Step 204: Using a large model, semantic extraction is performed on the prompts input by the user based on the structured financial certificate information to generate initial extracted content.

[0103] In this embodiment of the invention, the user inputs a prompt describing the extraction target and output format according to specific business needs. For example, the prompt might be used to extract the company name and return it in JSON format as {"company": "xxx"}. The user then inputs the structured financial certificate information and the prompt into the large model. The large model uses its semantic understanding and reasoning capabilities to locate and extract the required elements from the structured information and generate the initial extracted content.

[0104] Step 205: Perform syntax validation on the initial extracted content. If the syntax validation passes, proceed to step 206; if the syntax validation fails, proceed to step 204.

[0105] In this embodiment of the invention, the output of the large model may have format errors, so the initial extracted content needs to be syntax-checked to see if the output format of the initial extracted content conforms to the JSON syntax format. If the check passes, it means that the output format of the initial extracted content is JSON format, and step 206 continues; if the check fails, it means that the output format of the initial extracted content is not JSON format, and a new request is sent to the large model for secondary inference, and step 204 continues.

[0106] Step 206: Using a preset rule base and regular expressions, map and perform regular expression validation on the initial extracted content based on the structured financial certificate information. If the mapping and regular expression validation pass, proceed to step 207; if the mapping or regular expression validation fails, proceed to step 208.

[0107] In this embodiment of the invention, the accuracy and consistency of key-value pairs are verified by using a rule base and regular expressions. A mapping relationship is set for the key (K), and the value (V) is compared and validated using regular expressions to correct errors in format and content, ensuring the standardization and accuracy of the final returned result.

[0108] Because the key names output by the large model may contain synonyms or variations (e.g., outputting "corporation" instead of "company"), the system will use a pre-defined synonym mapping table (e.g., {"corporation": "company", "enterprise": "company"}) to uniformly map them to standard key names. The key mapping verification specifically involves determining whether the key name of the initially extracted content is a standard key name in the synonym mapping table. If it is, the mapping verification passes; otherwise, the mapping verification fails.

[0109] Value regular expression validation specifically involves using regular expressions to validate the format of extracted values. For example, it can validate whether the amount conforms to a numeric format, whether the date is in the correct format, and whether the character length is within a reasonable range. If all validation sub-items pass, the regular expression validation is successful; if at least one validation sub-item fails, the regular expression validation fails.

[0110] In this embodiment of the invention, if both the mapping verification and the regular expression verification pass, step 207 is executed; if the mapping verification fails or the regular expression verification fails, it indicates that the content may have illusions or formatting issues, and step 208 is executed.

[0111] Step 207: Determine the initial extracted content as the target extracted content, and proceed to step 209.

[0112] In this embodiment of the invention, if both the mapping and regular expression verification pass, the initial extracted content is considered accurate and usable, and the initial extracted content is determined as the target extracted content, and step 209 is continued.

[0113] Step 208: Based on the structured financial certificate information, repair the initial extraction content and generate the target extraction content.

[0114] In this embodiment of the invention, if mapping verification or regular expression verification fails, a repair mechanism is activated. The structured financial certificate information is used as a trusted data source to correct the failed values. For example, if the amount extracted by the large model is inconsistent with the amount in the structured financial certificate information, the result from the structured financial certificate information is used for replacement.

[0115] Furthermore, regular expressions can be used to remove spaces, concatenate, and perform secondary extraction to obtain the target content.

[0116] Furthermore, by adding samples whose results of structured financial certificate information are aligned with the extraction results to the fine-tuning data of the large model, the large model is optimized to improve output consistency.

[0117] This invention constructs an automated verification and repair closed loop through syntax validation, key-value mapping rules, and content review based on structured information. It can effectively identify and correct format errors, key name ambiguities, and content deviations in the output of large models, transforming the original output of large models into highly standardized, accurate, and reliable business data, and significantly improving the accuracy and direct usability of the final results.

[0118] Step 209: Perform regular expression post-processing on the target extracted content to generate the processed target extracted content.

[0119] In this embodiment of the invention, post-processing of regular expressions includes, but is not limited to: removing extra spaces at the beginning and end of the string, formatting specific fields (such as adding thousands separators to amounts), and concatenating fields that are broken due to recognition errors. After processing, the final processed target content that can be delivered to downstream business systems is generated to unify the format and improve readability.

[0120] For example: Change "xxx Company" to "xxx Company".

[0121] The large model input of this invention includes layout and table structure information, which is more suitable for the extraction needs of financial document images and retains structured information. The post-processing mechanism corrects errors that may be generated by the large model, ensuring the accuracy and stability of the system, improving post-processing assurance, and effectively improving the accuracy, efficiency and applicability of financial document information extraction, providing strong technical support for automated processing in the financial field.

[0122] The technical solution of the financial document image content extraction method based on a large model provided in this invention involves extracting elements and performing structured processing on the acquired financial document image to generate structured financial document information. Using a large model, semantic extraction is performed on the prompts input by the user based on the structured financial document information to generate initial extracted content. Based on the structured financial document information, the initial extracted content is verified and repaired to generate target extracted content. Through deep structured processing, the complex layout and table logic in the financial document image are accurately analyzed, ensuring the completeness of information extraction. Furthermore, semantic extraction is performed using a large model, and an automated verification and repair mechanism based on structured information effectively ensures the high accuracy and format consistency of the output results, significantly improving the automation level, adaptability, and reliability of processing various types of financial documents.

[0123] This invention also provides a device for extracting the content of financial document images based on a large model, as described in the following embodiments. Since the principle by which this device solves the problem is similar to that of the method for extracting the content of financial document images based on a large model, the implementation of this device can refer to the implementation of the method for extracting the content of financial document images based on a large model; repeated details will not be elaborated further.

[0124] Figure 3 This is a schematic diagram of a financial document image content extraction device based on a large model, provided in an embodiment of the present invention. This device is used to execute the aforementioned financial document image content extraction method based on a large model, such as... Figure 3 As shown, the device includes: an element structuring processing unit 11, a semantic extraction unit 12, and a content extraction unit 13.

[0125] The element structuring processing unit 11 is used to extract elements and perform structuring processing on the acquired financial certificate image to generate structured financial certificate information.

[0126] The semantic extraction unit 12 is used to extract semantics from the prompt words input by the user through a large model based on structured financial certificate information, and generate initial extracted content.

[0127] The content extraction unit 13 is used to perform verification and repair processing on the initial extracted content based on structured financial certificate information, and generate the target extracted content.

[0128] In this embodiment of the invention, the element structuring processing unit 11 is specifically used to perform orientation correction processing on the financial certificate image to generate a corrected financial certificate image; extract elements from the corrected financial certificate image using a pre-trained layout analysis model to generate certificate element information; and perform structuring processing on the certificate element information to generate structured financial certificate information.

[0129] In this embodiment of the invention, the element structuring processing unit 11 is specifically used to perform text line detection on the financial certificate image using a text detection algorithm to generate candidate text line regions; to perform direction recognition on the candidate text line regions using a lightweight classification model to generate multiple candidate directions; to determine the overall text direction based on the multiple candidate directions; and to perform direction correction on the financial certificate image based on the overall text direction to generate a corrected financial certificate image.

[0130] In this embodiment of the invention, the device further includes: a layout analysis model training unit 14.

[0131] The layout analysis model training unit 14 is used to train the target detection algorithm based on the pre-acquired document layout dataset to build a layout analysis model.

[0132] In this embodiment of the invention, the element structuring processing unit 11 is specifically used to perform multi-scale feature detection on the corrected financial certificate image to generate candidate layout blocks and corresponding element coordinates; classify the feature vectors of the candidate layout blocks to generate element classifications; perform content recognition on different element classifications to obtain the corresponding element content; and perform post-processing based on element coordinates, element classifications and element content to generate certificate element information.

[0133] In this embodiment of the invention, the content extraction unit 13 is used to perform syntax verification on the initial extracted content; if the syntax verification passes, the initial extracted content is mapped and verified using a preset rule base and regular expressions based on structured financial certificate information; if the mapping and regular expression verification pass, the initial extracted content is determined as the target extracted content; if the mapping or regular expression verification fails, the initial extracted content is repaired based on the structured financial certificate information to generate the target extracted content; if the syntax verification fails, the semantic extraction unit 12 is triggered to repeatedly execute the steps of semantic extraction of prompt words input by the user based on the structured financial certificate information through the large model to generate the initial extracted content.

[0134] In this embodiment of the invention, the device further includes: a regularization post-processing unit 15.

[0135] The regular expression post-processing unit 15 is used to perform regular expression post-processing on the target extracted content to generate the processed target extracted content.

[0136] In this embodiment of the invention, the acquired financial document image undergoes element extraction and structuring processing to generate structured financial document information. A large model is used to semantically extract user-input prompts based on the structured financial document information, generating initial extracted content. Based on the structured financial document information, the initial extracted content is validated and repaired to generate target extracted content. Through deep structuring processing, the complex layout and table logic in the financial document image are accurately analyzed, ensuring the completeness of information extraction. Furthermore, semantic extraction is performed using a large model, and an automated validation and repair mechanism based on structured information effectively ensures high accuracy and format consistency of the output results, significantly improving the automation level, adaptability, and reliability of processing various financial documents.

[0137] It should be noted that the financial document image content extraction method based on large models provided in this embodiment of the invention can be used in the financial field, or in any technical field other than the financial field. This embodiment of the invention does not limit the application field of the financial document image content extraction method based on large models.

[0138] Figure 4 This is a schematic diagram of the physical structure of a computer device provided in an embodiment of the present invention, such as... Figure 4 As shown, the computer device includes a processor 501, a memory 502, and a bus 503.

[0139] The processor 501 and the memory 502 communicate with each other via the bus 503.

[0140] The processor 501 is used to call program instructions in the memory 502 to execute the methods provided in the above-described method embodiments.

[0141] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for extracting financial document image content based on a large model.

[0142] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for extracting financial document image content based on a large model.

[0143] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0144] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0145] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0146] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0147] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for extracting content from financial document images based on a large model, characterized in that, include: The acquired financial document images are subjected to element extraction and structuring processing to generate structured financial document information; Using a large model, semantic extraction is performed on the prompt words input by the user based on the structured financial certificate information to generate initial extracted content; Based on the structured financial certificate information, the initial extracted content is verified and repaired to generate the target extracted content.

2. The method for extracting financial document image content based on a large model according to claim 1, characterized in that, The step of extracting elements and performing structuring processing on the acquired financial document image to generate structured financial document information includes: The orientation correction process is performed on the financial document image to generate a corrected financial document image; The corrected financial certificate image is extracted using a pre-trained layout analysis model to generate certificate element information. The information of the voucher elements is processed in a structured manner to generate structured financial voucher information.

3. The method for extracting financial document image content based on a large model according to claim 2, characterized in that, The step of performing orientation correction processing on the financial document image to generate a corrected financial document image includes: The financial document image is subjected to text line detection using a text detection algorithm to generate candidate text line regions. A lightweight classification model is used to identify the direction of the candidate text line regions, generating multiple candidate directions; The overall direction of the text is determined based on multiple candidate directions; Based on the overall orientation of the text, the orientation of the financial certificate image is corrected to generate a corrected financial certificate image.

4. The method for extracting financial document image content based on a large model according to claim 2, characterized in that, The method further includes: Based on a pre-acquired document layout dataset, the target detection algorithm is trained to construct the layout analysis model.

5. The method for extracting financial document image content based on a large model according to claim 2, characterized in that, The step of extracting elements from the corrected financial document image using a pre-trained layout analysis model to generate document element information includes: Multi-scale feature detection is performed on the corrected financial document image to generate candidate layout blocks and corresponding element coordinates; The feature vectors of the candidate page blocks are classified to generate element classifications; Content recognition is performed on different element categories to obtain the corresponding element content; The voucher element information is generated by post-processing based on the element coordinates, element classification, and element content.

6. The method for extracting financial document image content based on a large model according to claim 1, characterized in that, The step of verifying and repairing the initial extracted content based on the structured financial certificate information to generate the target extracted content includes: Perform syntax validation on the initially extracted content; If the syntax validation passes, the initial extracted content is mapped and validated using a preset rule base and regular expressions based on the structured financial certificate information. If the mapping and regular expression validation pass, the initial extracted content will be determined as the target extracted content; If the mapping or regular expression validation fails, the initial extracted content is repaired based on the structured financial certificate information to generate the target extracted content; If the syntax check fails, repeat the step of using a large model to semantically extract the prompt words input by the user based on the structured financial certificate information to generate the initial extracted content.

7. The method for extracting financial document image content based on a large model according to claim 1, characterized in that, The method further includes: The extracted target content is post-processed using regular expressions to generate the processed extracted target content.

8. A device for extracting content from financial document images based on a large model, characterized in that, include: The element structuring processing unit is used to extract elements and perform structuring processing on the acquired financial certificate images to generate structured financial certificate information. The semantic extraction unit is used to extract semantics from the prompt words input by the user based on the structured financial certificate information using a large model, and generate initial extracted content. The content extraction unit is used to perform verification and repair processing on the initial extracted content based on the structured financial certificate information, and generate the target extracted content.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images