Financial statement processing method and device based on multi-modal large model

Abstract

According to the financial statement processing method and device based on the multi-modal large model, the to-be-processed financial statement image is obtained and preprocessed, the to-be-recognized image after quality optimization is obtained, and the recognition accuracy is effectively improved; the to-be-recognized image is input into the multi-modal large model, a structured recognition result which is output by the multi-modal large model and comprises at least one key value pair is obtained, the multi-modal large model can recognize financial statements in various formats, and the adaptability is high; the structured recognition result is compared with the preset verification rule, compliance verification based on financial logic is executed, and through automatic intelligent verification, the dependence on manpower is reduced, and the business efficiency is greatly improved; under the condition that the verification result indicates that the structured recognition result is abnormal, linkage display is provided on a front-end interface, and correction data input by a user is acquired; and finally, generating final structured financial data on the basis of an identification result passing the verification or on the basis of the correction data.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a financial statement processing method and apparatus based on a multimodal large model. Background Technology

[0002] In the daily operations of banks, financial institutions, and enterprises, a massive amount of financial statements, such as balance sheets, income statements, and cash flow statements, need to be processed daily. Currently, the mainstream method for processing this unstructured financial information still heavily relies on manual visual input and cross-system manual verification, which presents the following urgent technical problems that need to be solved: 1. Issues of Format Diversity and Non-Standardization: Report formats vary greatly among different enterprises, industries, and accounting standards. Traditional OCR (Optical Character Recognition) technology relies heavily on fixed templates, making it poorly adaptable to newly emerging or variant report formats and unable to achieve generalized recognition, resulting in high system expansion and maintenance costs.

[0003] 2. Low recognition accuracy: When faced with complex scenarios such as blurry scanned documents, incomplete table lines, or handwritten annotations, the recognition error rate of traditional OCR engines remains high, which seriously affects the reliability of subsequent data use.

[0004] 3. Lack of intelligent verification loop: Traditional processes stop at text extraction and lack semantic-level business verification of the recognition results. For example, they cannot automatically verify whether the balance sheet is balanced or whether the relevant items in the income statement and cash flow statement are consistent. The correctness of key financial logic still needs to be judged manually, resulting in a low degree of automation.

[0005] 4. High cost of manual intervention: Each report requires an average of 2-3 employees to participate in the entire process of "identification-verification-correction-archiving". The processing time for a single business transaction exceeds 10 minutes, which seriously restricts business efficiency and is prone to secondary errors due to fatigue.

[0006] Therefore, while existing technologies can perform basic text extraction, they fall far short of achieving a complete, intelligent, and reusable automated closed loop from image input to business-usable structured data output. An innovative solution is urgently needed to overcome these shortcomings. Summary of the Invention

[0007] This invention provides a financial statement processing method and apparatus based on a multimodal large model, which solves the problems of poor format adaptability, low accuracy, lack of intelligent verification and high labor costs faced by traditional OCR technology in financial statement processing.

[0008] This invention provides a financial statement processing method based on a multimodal large model, comprising: Obtain the image of the financial statement to be processed; The financial statement image is preprocessed to obtain a quality-optimized image to be recognized; The image to be identified is input into a multimodal large model to obtain a structured recognition result containing at least one key-value pair, which is output by the multimodal large model after parsing the semantic structure of the image to be identified; wherein, the key-value pair includes the name of the financial subject and its corresponding value identified from the image to be identified; The structured recognition results are compared with preset verification rules to perform compliance verification based on financial logic; wherein, the verification rules define the reconciliation relationship between the values ​​of different financial items. If the verification result indicates that the structured recognition result is abnormal, the front-end interface will provide a linked display and obtain the correction data input by the user; Based on the verified identification results or based on the corrected data, the final structured financial data is generated.

[0009] According to the present invention, a financial statement processing method based on a multimodal large model is provided. The multimodal large model is a zero-shot inference model obtained by pre-training with multimodal data, which can parse the semantic structure of the image to be identified based on a general semantic understanding of financial statements.

[0010] According to the present invention, a financial statement processing method based on a multimodal large model is provided, wherein the multimodal large model is scheduled by a unified management platform; the management platform is used to manage multiple artificial intelligence models and assign appropriate artificial intelligence models to the financial statement images for processing according to a preset strategy.

[0011] According to the financial statement processing method based on a multimodal large model provided by the present invention, before acquiring the financial statement image to be processed, the method further includes: Obtain the pre-configured field structure and validation rules corresponding to the financial statements.

[0012] According to the financial statement processing method based on a multimodal large model provided by the present invention, after obtaining the corrected data input by the user, it further includes: The corrected data is associated with and stored in relation to the corresponding financial statement image to form a labeled sample; The labeled samples are used to fine-tune the training of the multimodal large model to optimize its recognition ability.

[0013] According to the present invention, a financial statement processing method based on a multimodal large model is provided, wherein the financial account names include account names under standard accounting standards and enterprise-defined account names; the method further includes: By leveraging the semantic understanding capabilities of the multimodal large model, the identified enterprise-defined account names are mapped to account codes under standard accounting standards.

[0014] According to the financial statement processing method based on a multimodal large model provided by the present invention, after generating the final structured financial data, it further includes: The final structured financial data is pushed to the downstream financial system in a predetermined format; wherein, the predetermined format includes JSON format, Excel format or CSV format.

[0015] The present invention also provides a financial statement processing device based on a multimodal large model, comprising: The data acquisition module is used to acquire images of financial statements to be processed. The preprocessing module is used to preprocess the financial statement image to obtain a quality-optimized image to be recognized. The structured recognition module is used to input the image to be recognized into a multimodal large model, and obtain a structured recognition result containing at least one key-value pair output by the multimodal large model after parsing the semantic structure of the image to be recognized; wherein, the key-value pair includes the name of the financial subject and its corresponding value identified from the image to be recognized; The rule verification module is used to compare the structured recognition results with preset verification rules and perform compliance verification based on financial logic; wherein, the verification rules define the reconciliation relationship between the values ​​of different financial items; The user interaction module is used to provide a linked display on the front-end interface and obtain the correction data input by the user when the verification result indicates that the structured recognition result is abnormal. The data generation module is used to generate final structured financial data based on the verified identification results or based on the corrected data.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the financial statement processing method based on a multimodal large model as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the financial statement processing method based on a multimodal large model as described above.

[0018] The financial statement processing method and apparatus based on a multimodal large model provided by this invention acquires an image of the financial statement to be processed; preprocesses the financial statement image to obtain a quality-optimized image to be recognized, effectively improving the recognition accuracy; inputs the image to be recognized into a multimodal large model to obtain a structured recognition result containing at least one key-value pair, output by the multimodal large model after parsing the semantic structure of the image to be recognized. The multimodal large model can recognize financial statements of various formats and has strong adaptability. The key-value pair includes the name of the financial item and its corresponding value identified from the image to be recognized; compares the structured recognition result with preset verification rules to perform compliance verification based on financial logic. Through automatic intelligent verification, the reliance on manual labor is reduced, and business efficiency is greatly improved. The verification rules define the correlation between the values ​​of different financial items; when the verification result indicates that the structured recognition result is abnormal, a linked display is provided on the front-end interface and corrective data input by the user is obtained; based on the verification-passed recognition result or based on the corrective data, the final structured financial data is generated. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the financial statement processing method based on a multimodal large model provided by the present invention. Figure 2 This is a schematic diagram illustrating the execution of an optional financial statement processing method based on a multimodal large model in an embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of the financial statement processing device based on a multimodal large model provided by the present invention; Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] The following is combined with Figure 1 and Figure 2 This invention describes a financial statement processing method based on a multimodal large model.

[0023] like Figure 1 As shown, the financial statement processing method based on a multimodal large model provided by this invention includes the following steps: S1. Obtain the image of the financial statement to be processed.

[0024] To provide convenient data configuration and verification rule settings, and to ensure the accuracy and compliance of recognition results, such as Figure 2 As shown, in an optional embodiment of the present invention, before acquiring the image of the financial statement to be processed, the method further includes: acquiring the pre-configured field structure and verification rules corresponding to the financial statement. Specifically, users can pre-configure the data management and verification rule engine, which is a module for configuring and managing the basic information and recognition rules of various financial statements. Users can configure the field structure, mapping code, sample image, and recognition model type (single model / multimodal large model) of each type of financial statement (such as balance sheet, income statement, cash flow statement) through a visual interface, and define cross-field verification rules based on accounting logic (trial balance) (such as "total assets = total liabilities + owner's equity", "net profit = operating revenue - operating costs - taxes and surcharges - selling expenses - administrative expenses - financial expenses"), providing a foundation for subsequent "recognition as verification, error as prompt". Through the data management and verification rule engine, users can easily configure new financial statement types and verification rules to ensure the accuracy and compliance of the recognition results and reduce manual intervention.

[0025] By using a configuration-based approach rather than hard-coding, the system can quickly adapt to new report formats. When encountering a new report format, simply configure a new "data" entry in the system; no code modification or model retraining is required, greatly improving the system's flexibility and scalability. Users can choose accounting standard templates and industry templates, or create custom templates, freely configuring "accounts" (batch adding / deleting standard accounts, adjusting sorting), "account mapping" (adding fixed mapping relationships), and "trial calculation formulas." For example, a bank needs to process Company A's custom-formatted profit and loss statement. Traditionally, this requires developing a new template, which is time-consuming and labor-intensive. In this solution, business personnel only need to: 1) upload a sample profit and loss statement from Company A; 2) select the location of key fields such as "operating revenue," "operating costs," and "net profit" on the sample image or define their semantics; 3) set validation rules such as "net profit = operating revenue - total operating costs." After configuration, the system can automatically process all subsequent similar reports submitted by Company A.

[0026] To provide efficient batch processing capabilities, in one optional embodiment of the present invention, the system supports batch uploading and processing of financial statements. It automatically decompresses compressed files and identifies individual financial statements within the compressed files, processing them sequentially. This is suitable for large-scale financial statement processing scenarios, reducing manual operations and improving processing efficiency. Furthermore, to facilitate problem tracking and performance optimization, and to help operations and maintenance personnel understand system operation, in one optional embodiment of the present invention, the system can also fully record the transaction log of each API call (including requester transaction number, response time, status code, etc.); and provide multi-dimensional call statistics dashboards by channel, engine, and data type, supporting operations and maintenance optimization and cost analysis.

[0027] S2. Preprocess the financial statement image to obtain an optimized image to be recognized.

[0028] Specifically, the uploaded financial statement images undergo preliminary processing, including but not limited to at least one of the following: exposure adjustment, wide dynamic range adjustment, white balance correction, and image enhancement (such as noise reduction, contrast enhancement, and edge sharpening). This ensures optimal image quality for subsequent manual review and understanding, while also improving the accuracy of OCR recognition. For example, a user uploads a balance sheet image taken with a mobile phone, which has shadows and reflections. The preprocessing module automatically corrects colors, enhances contrast, and sharpens text edges, generating a clear image close to that of a flatbed scan. This image is then sent to the recognition module, significantly improving the accuracy of text and table line recognition.

[0029] S3. Input the image to be recognized into the multimodal large model, and obtain the structured recognition result containing at least one key-value pair output by the multimodal large model after parsing the semantic structure of the image to be recognized.

[0030] Specifically, a multimodal large model is a deep learning model capable of processing various types of data (such as text, images, and audio) and extracting high-level feature representations. In this invention, a multimodal large model is used for end-to-end zero-shot inference, automatically extracting key information from financial statements (such as automatically extracting key-value pairs: "Total Assets: ¥1,250,000", "Net Profit: ¥250,000", etc.) and generating structured recognition results. Zero-shot inference refers to the model's ability to accurately identify and classify new and unknown samples even without having seen or trained on specific samples. Traditional OCR relies on fixed templates and cannot handle new formats. In this invention, the multimodal large model, through zero-shot inference technology, can automatically recognize non-standard and new-format financial statements. Even without having seen a certain statement format before, it can accurately extract information based on a general semantic understanding of financial statements (such as knowing that a balance sheet necessarily contains concepts like "assets" and "liabilities"), without needing to pre-train specific templates, significantly reducing model iteration costs. The system can process financial statements in various unknown formats, achieving efficient identification of non-standard financial statements, reducing reliance on specific templates, and improving the system's flexibility and generalization capabilities. Furthermore, the multimodal large model exhibits significantly higher robustness to complex scenarios such as fuzzy, distorted, and handwritten annotations compared to traditional OCR. It can reason based on context; for example, even if the words "current assets" are somewhat blurry, the model can infer the field name with a high probability based on its location and surrounding fields such as "cash and cash equivalents" and "accounts receivable."

[0031] Key-value pairs include the names of financial accounts identified from the image to be recognized and their corresponding values. The names of financial accounts include account names under standard accounting standards and enterprise-defined account names. In an optional embodiment of the present invention, the above-described financial statement processing method based on a multimodal large model further includes: utilizing the semantic understanding capabilities of the multimodal large model to map the identified enterprise-defined account names to account codes under standard accounting standards. Specifically, relying on the capabilities of the large model and semantic understanding technology, the identified enterprise-defined accounts (such as "inventory materials") can be automatically mapped to system-preset (under standard accounting standards) standard accounts (such as "inventory"), significantly reducing the workload of manual matching caused by inconsistent account names, and solving the mapping problem of non-standard accounts through semantic understanding.

[0032] In existing related technologies, each business system typically purchases or develops its own OCR capabilities (such as table recognition and text recognition), lacking unified management, resulting in low algorithm reuse rates and high maintenance costs. To address this issue, in an optional embodiment of this invention, multimodal large models are scheduled by a unified management platform. The management platform manages multiple artificial intelligence models and assigns appropriate artificial intelligence models to financial statement images for processing according to preset strategies. The management platform can uniformly register, manage versions, monitor access, and schedule resources for all OCR and AI capabilities within the system (including fixed-format OCR, general text recognition, general table recognition, seal recognition, image classification, etc.). This platform integrates OCR and AI capabilities from different sources to form a reusable, replaceable, and scalable "AI capability supermarket," and supports the management of third-party OCR engines (such as SenseTime, Baidu, iFlytek, etc.), enabling dynamic switching and canary release of capabilities. This achieves centralized management and flexible scheduling of AI capabilities, improving system scalability and maintainability, and avoiding redundant construction and resource waste.

[0033] S4. Compare the structured recognition results with the preset verification rules and perform compliance verification based on financial logic.

[0034] The verification rules define the reconciliation relationships between different financial account values. In accounting, a reconciliation relationship refers to the mutual verification and cross-checking between relevant figures in ledgers and financial statements. For example, the ending balance of a category account must match the sum of its corresponding subsidiary accounts, and the amounts of similar items in the income statement and the product sales statement must match.

[0035] Before using the system, users can pre-configure verification rules for various reports through a visual interface. These rules define the reconciliation relationships between financial items, the most typical being the trial balance rule for the balance sheet: Total Assets = Total Liabilities + Total Owner's Equity. The rule verification module automatically executes these rules. This step advances the process from "recognition" to "understanding and verification," transforming business knowledge into automated rules for secondary verification of the recognition results. This is no longer simple character checking, but a verification of the correctness of data logic, automatically identifying deep-seated logical contradictions in the recognition results. For example, when three identified values ​​do not satisfy the balance formula, the system immediately marks an anomaly. Automated verification can detect most logical errors, freeing manual labor from "comprehensive checking" to "targeted correction," significantly improving efficiency. Simultaneously, it can discover deep-seated errors that the recognition module itself may not be able to detect.

[0036] S5. If the verification result indicates that the structured recognition result is abnormal, provide a linked display on the front-end interface and obtain the correction data input by the user.

[0037] Specifically, this invention displays the recognition results and the original image on the front-end interface. Users can click on fields in the recognition results, and the corresponding areas on the original image will be highlighted, facilitating quick location and verification. Furthermore, the front-end interface provides an Excel-like editing experience, allowing users to directly modify the recognition results. Through excellent human-computer interaction design, the correction process becomes highly efficient. The linked highlighting function eliminates the time spent manually comparing images and result lists repeatedly, minimizing the time required for a single correction. For example, if the verification engine indicates low confidence in the "Operating Revenue" field, the user can click on the field, and the system will automatically highlight the row containing "Operating Revenue" on the original image. If the user finds that it has been correctly identified as "Operating Income," they can simply enter the correct value in the edit box; the entire process may only take a few seconds.

[0038] To continuously optimize the model's recognition capabilities, such as Figure 2 As shown, in an optional embodiment of the present invention, after obtaining the user-inputted correction data, the method further includes: associating and storing the correction data with the corresponding financial statement image to form labeled samples; and using the labeled samples to fine-tune and train the multimodal large model to optimize its recognition ability. By automatically recording and feeding back the manually corrected high-quality data into the training data pool, and using new high-quality data to periodically fine-tune or optimize the multimodal large model, the model acquires the ability to become more accurate with use. For example, if a particular seal of a company is frequently misidentified, after several manual corrections, the model will learn this feature, and the recognition accuracy of the company's reports will naturally improve subsequently.

[0039] S6. Based on the verified identification results or based on the corrected data, generate the final structured financial data.

[0040] By providing standardized data output, it facilitates the integration and use of downstream systems (such as ERP and credit risk control systems), ensuring data consistency and readability.

[0041] In an optional embodiment of the present invention, after generating the final structured financial data, the method further includes: pushing the final structured financial data to the downstream financial system in a predetermined format; wherein the predetermined format includes, but is not limited to, JSON format, Excel format, or CSV format.

[0042] The financial statement processing method based on a multimodal large model provided by this invention has been implemented and verified in multiple bank POC projects, achieving the following quantifiable technical effects: Table 1. Comparison of the effects of the present invention and traditional solutions.

[0043] The financial statement processing method based on a multimodal large model provided by this invention has achieved significant breakthroughs in terms of recognition accuracy, processing efficiency, and data quality: Significantly improved recognition accuracy: For non-standard and complex format reports, the recognition accuracy has increased from the traditional 68%~75% to 92%~96%, significantly reducing the amount of manual correction work.

[0044] A qualitative leap in processing efficiency: The processing time for a single-page report has been reduced from 20-30 minutes to less than 1 minute; and it supports automatic splitting and classification of complex multi-page documents, with the entire process taking only 23 minutes, realizing the transformation from "manual search" to "fully automatic extraction".

[0045] Intelligent subject matching: It uses large-scale model semantic understanding to replace manual mapping, and the accuracy rate of standard subject matching reaches more than 95%, effectively solving the problem of unifying heterogeneous reports.

[0046] Significantly improved data quality: Through the built-in trial balance verification mechanism, the data error rate of downstream applications has been reduced from 12%~18% to below 3%, ensuring the accuracy and compliance of financial data.

[0047] High reusability reduces costs: Based on a unified management platform, the reusability of AI capabilities has been increased from less than 30% to 85%, significantly reducing the development and maintenance costs of new business scenarios.

[0048] In summary, the financial statement processing method based on a multimodal large model provided by this invention achieves zero-sample understanding through a multimodal large model, eliminating the need to train a model for each new report, significantly reducing model iteration costs, and solving the problem of traditional OCR's "inability to recognize and extend" non-standard financial statements. Unified management enables capability reuse, resource optimization, and centralized operation and maintenance, reducing system coupling and improving scalability, thus addressing the architectural pain points of existing technologies where AI capabilities are "independent and repeatedly deployed." Through an integrated closed loop of "recognition + rule verification + online revision," manual intervention is reduced from "full-process participation" to "error correction," solving the inefficiency and high cost of manual full-process verification in existing technologies. By recording each manual correction, a high-quality labeled data pool is formed, feeding back into the model for continuous optimization, achieving intelligent evolution that becomes "more accurate with use." Through interface configuration of "data + rules," users can quickly deploy new business scenarios (such as adding "rural land contract certificate" recognition) without programming, reducing response time from weekly to hourly, greatly improving business adaptability and efficiency.

[0049] Based on the same inventive concept, the present invention also provides a financial statement processing device based on a multimodal large model. The financial statement processing device based on a multimodal large model provided by the present invention will be described below. The financial statement processing device based on a multimodal large model described below and the financial statement processing method based on a multimodal large model described above can be referred to in correspondence.

[0050] like Figure 3 As shown, the financial statement processing device based on a multimodal large model provided by the present invention includes a data acquisition module 31, a preprocessing module 32, a structured recognition module 33, a rule verification module 34, a user interaction module 35, and a data generation module 36.

[0051] Data acquisition module 31 is used to acquire images of financial statements to be processed; Preprocessing module 32 is used to preprocess the financial statement image to obtain a quality-optimized image to be recognized; The structured recognition module 33 is used to input the image to be recognized into a multimodal large model and obtain a structured recognition result containing at least one key-value pair output by the multimodal large model after parsing the semantic structure of the image to be recognized; wherein, the key-value pair includes the name of the financial subject and its corresponding value identified from the image to be recognized; The rule verification module 34 is used to compare the structured recognition result with the preset verification rules and perform compliance verification based on financial logic; wherein, the verification rules define the reconciliation relationship between the values ​​of different financial items; User interaction module 35 is used to provide a linked display on the front-end interface and obtain correction data input by the user when the verification result indicates that the structured recognition result is abnormal. The data generation module 36 is used to generate the final structured financial data based on the verified identification results or based on the corrected data.

[0052] In an optional embodiment of the present invention, the above-described financial statement processing device based on a multimodal large model further includes a rule definition module and an optimization iteration module.

[0053] The rule definition module is used to obtain the field structure and validation rules corresponding to the pre-configured financial statements.

[0054] The optimization iteration module is used to associate and store the corrected data with the corresponding financial statement image to form labeled samples; and to fine-tune and train the multimodal large model using the labeled samples to optimize its recognition ability.

[0055] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute the financial statement processing method based on a multimodal large model provided by the above methods, which includes: Obtain the image of the financial statement to be processed; The financial statement image is preprocessed to obtain a quality-optimized image to be recognized; The image to be identified is input into a multimodal large model to obtain a structured recognition result containing at least one key-value pair, which is output by the multimodal large model after parsing the semantic structure of the image to be identified; wherein, the key-value pair includes the name of the financial subject and its corresponding value identified from the image to be identified; The structured recognition results are compared with preset verification rules to perform compliance verification based on financial logic; wherein, the verification rules define the reconciliation relationship between the values ​​of different financial items. If the verification result indicates that the structured recognition result is abnormal, the front-end interface will provide a linked display and obtain the correction data input by the user; Based on the verified identification results or based on the corrected data, the final structured financial data is generated.

[0056] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0057] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the financial statement processing method based on a multimodal large model provided by the above methods, the method comprising: Obtain the image of the financial statement to be processed; The financial statement image is preprocessed to obtain a quality-optimized image to be recognized; The image to be identified is input into a multimodal large model to obtain a structured recognition result containing at least one key-value pair, which is output by the multimodal large model after parsing the semantic structure of the image to be identified; wherein, the key-value pair includes the name of the financial subject and its corresponding value identified from the image to be identified; The structured recognition results are compared with preset verification rules to perform compliance verification based on financial logic; wherein, the verification rules define the reconciliation relationship between the values ​​of different financial items. If the verification result indicates that the structured recognition result is abnormal, the front-end interface will provide a linked display and obtain the correction data input by the user; Based on the verified identification results or based on the corrected data, the final structured financial data is generated.

[0058] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the financial statement processing method based on a multimodal large model provided by the methods described above, the method comprising: Obtain the image of the financial statement to be processed; The financial statement image is preprocessed to obtain a quality-optimized image to be recognized; The image to be identified is input into a multimodal large model to obtain a structured recognition result containing at least one key-value pair, which is output by the multimodal large model after parsing the semantic structure of the image to be identified; wherein, the key-value pair includes the name of the financial subject and its corresponding value identified from the image to be identified; The structured recognition results are compared with preset verification rules to perform compliance verification based on financial logic; wherein, the verification rules define the reconciliation relationship between the values ​​of different financial items. If the verification result indicates that the structured recognition result is abnormal, the front-end interface will provide a linked display and obtain the correction data input by the user; Based on the verified identification results or based on the corrected data, the final structured financial data is generated.

[0059] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0060] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A financial statement processing method based on a multimodal large model, characterized in that, include: Obtain the image of the financial statement to be processed; The financial statement image is preprocessed to obtain a quality-optimized image to be recognized; The image to be identified is input into a multimodal large model to obtain a structured recognition result containing at least one key-value pair, which is output by the multimodal large model after parsing the semantic structure of the image to be identified; wherein, the key-value pair includes the name of the financial subject and its corresponding value identified from the image to be identified; The structured recognition results are compared with preset verification rules to perform compliance verification based on financial logic; wherein, the verification rules define the reconciliation relationship between the values ​​of different financial items. If the verification result indicates that the structured recognition result is abnormal, the front-end interface will provide a linked display and obtain the correction data input by the user; Based on the verified identification results or based on the corrected data, the final structured financial data is generated.

2. The financial statement processing method based on a multimodal large model according to claim 1, characterized in that, The multimodal large model is a zero-shot inference model obtained through multimodal data pre-training, which can parse the semantic structure of the image to be identified based on the general semantic understanding of financial statements.

3. The financial statement processing method based on a multimodal large model according to claim 2, characterized in that, The multimodal large model is scheduled by a unified management platform; the management platform is used to manage multiple artificial intelligence models and assign appropriate artificial intelligence models to the financial statement images for processing according to preset strategies.

4. The financial statement processing method based on a multimodal large model according to claim 1, characterized in that, Before obtaining the image of the financial statement to be processed, the following steps are also included: Obtain the pre-configured field structure and validation rules corresponding to the financial statements.

5. The financial statement processing method based on a multimodal large model according to claim 1, characterized in that, After obtaining the correction data input by the user, it also includes: The corrected data is associated with and stored in relation to the corresponding financial statement image to form a labeled sample; The labeled samples are used to fine-tune the training of the multimodal large model to optimize its recognition ability.

6. The financial statement processing method based on a multimodal large model according to claim 1, characterized in that, The financial account names include account names under standard accounting standards and enterprise-defined account names; the method also includes: By leveraging the semantic understanding capabilities of the multimodal large model, the identified enterprise-defined account names are mapped to account codes under standard accounting standards.

7. The financial statement processing method based on a multimodal large model according to any one of claims 1 to 6, characterized in that, After generating the final structured financial data, the following is also included: The final structured financial data is pushed to the downstream financial system in a predetermined format; wherein, the predetermined format includes JSON format, Excel format or CSV format.

8. A financial statement processing device based on a multimodal large model, characterized in that, include: The data acquisition module is used to acquire images of financial statements to be processed. The preprocessing module is used to preprocess the financial statement image to obtain a quality-optimized image to be recognized. The structured recognition module is used to input the image to be recognized into a multimodal large model, and obtain a structured recognition result containing at least one key-value pair output by the multimodal large model after parsing the semantic structure of the image to be recognized; wherein, the key-value pair includes the name of the financial subject and its corresponding value identified from the image to be recognized; The rule verification module is used to compare the structured recognition results with preset verification rules and perform compliance verification based on financial logic; wherein, the verification rules define the reconciliation relationship between the values ​​of different financial items; The user interaction module is used to provide a linked display on the front-end interface and obtain the correction data input by the user when the verification result indicates that the structured recognition result is abnormal. The data generation module is used to generate final structured financial data based on the verified identification results or based on the corrected data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the financial statement processing method based on a multimodal large model as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the financial statement processing method based on a multimodal large model as described in any one of claims 1 to 7.

Images