A Method and Related Device for Parsing Financial Instructions Based on Multimodal Large Model

Abstract

This application provides a method and related apparatus for parsing financial instructions from images based on a multimodal large model. The method includes: S1, inputting an original document image containing investment transaction instructions, and using an optical character recognition (OCR) model to perform text recognition on the original document image to obtain a list of text blocks; S2, for the text block list, constructing image transaction instruction parsing prompts and location information vectors, and inputting the image transaction instruction parsing prompts and location information vectors into an image transaction instruction parsing large model, which outputs a corresponding list of transaction instructions, the list of transaction instructions containing all transaction instructions in the image; S3, comparing and verifying the list of transaction instructions with a transaction business knowledge base, supplementing auxiliary elements according to business rules, and correcting text errors caused by OCR errors based on text features; S4, outputting the complete transaction instructions after the above processing steps.

Description

Technical Field

[0001] This invention relates to the field of image text element recognition, specifically to a method and related apparatus for parsing financial instructions from images based on a multimodal large model. Background Technology

[0002] In the digital transformation of the investment field, the recognition and parsing of image-based trading instructions has become a key link in improving trading efficiency. Traditionally, instructions from trading managers to traders often appear in the form of images, which contain a wealth of trading information, including key data such as stock codes, trading volumes, and prices.

[0003] However, image commands are often accompanied by complex backgrounds, blurry text, or industry-specific symbols and terminology, which poses a significant challenge to the accurate understanding and rapid execution of the commands. Summary of the Invention

[0004] To address this challenge, this application provides a method, system, computer-readable storage medium, and apparatus for parsing financial instructions from images based on a multimodal large-scale model. This method, utilizing deep learning technology, can accurately extract trading instruction information from images, maintaining high recognition accuracy even in complex backgrounds or with blurred text. The system leverages the pre-training capabilities of a multimodal large-scale language model, combined with natural language processing and image recognition techniques. Through alignment pre-training and supervised fine-tuning, it enhances the understanding of investment-related terminology and symbols, ensuring the accuracy of instruction parsing.

[0005] The application of this innovative method has brought about significant changes in the investment field. First, the automatic image instruction parsing technology can substantially reduce the labor costs of fund companies, shorten transaction cycles, and significantly improve work efficiency. Second, through the powerful parameter advantages of large language models, the system can more accurately understand the information in image instructions, reducing reliance on manual intervention and improving the accuracy and efficiency of transactions. Finally, through linkage with enterprise knowledge bases, the system can further optimize the post-processing of image instructions, achieving rule verification and matching in business logic, ensuring the compliance and security of transactions.

[0006] In conclusion, the image-based trading instruction recognition technology based on a multimodal large language model not only represents the latest advancement in automation technology in the investment field, but also provides strong technical support for improving trading efficiency, reducing costs, and ensuring trading accuracy, marking an important milestone in the intelligent transformation of the investment field.

[0007] This invention aims to overcome the efficiency and accuracy challenges of traditional manual image-based trading instruction parsing. In traditional parsing processes, relying on manual reading and understanding of image instructions not only consumes significant time and human resources but is also highly susceptible to subjective judgment, increasing the potential risk of parsing errors. Faced with ever-increasing trading volumes and a complex and volatile market environment, the investment field urgently needs a faster and more intelligent solution to significantly improve the efficiency and accuracy of instruction recognition.

[0008] The image trading instruction parsing method and system based on a multimodal large language model proposed in this invention can automate and intelligently parse instructions, significantly accelerating parsing speed, greatly reducing reliance on manual labor, and thus significantly improving work efficiency. The introduction of the large language model not only handles image instructions in complex contexts but also understands industry-specific symbols and terminology through deep learning, reducing parsing errors and ensuring accurate execution of trading instructions. This innovative solution marks a solid step forward in the automation and intelligence of the investment field, providing strong technical support for improving trading efficiency, reducing operating costs, and enhancing market competitiveness.

[0009] In view of this, the first aspect of this application provides a method for parsing financial instructions from images based on a multimodal large model. The method includes: Step S1, inputting an original document image containing investment transaction instructions, and using an optical character recognition (OCR) model to perform text recognition on the original document image to obtain a list of text blocks containing text content and text coordinates; Step S2, for the text block list, constructing image transaction instruction parsing prompt statements and location information vectors, and inputting the image transaction instruction parsing prompt statements and location information vectors into an image transaction instruction parsing large model, which outputs a corresponding list of transaction instructions, the list of transaction instructions containing all transaction instructions in the image; Step S3, for the list of transaction instructions, comparing and verifying each instruction with a transaction business knowledge base, supplementing auxiliary elements according to business rules, and correcting text errors caused by OCR errors based on text features to ensure its completeness and accuracy; Step S4, outputting the complete transaction instruction after the above processing steps.

[0010] In conjunction with the first aspect, in one possible implementation, step S2 includes: Step S21, for the text content in the text block list, constructing an investment transaction instruction image recognition prompt statement, wherein the investment transaction instruction image recognition prompt statement includes one or more of the following: A. Task description prompt statement, B. Target recognition element name list prompt statement, C. Element example prompt statement, D. Output format example prompt statement, E. Investment transaction instruction image text content to be parsed; wherein, the task description prompt statement is used to explain the text source and output format of the recognition task target; the target recognition element name list prompt statement is used to display the element names in a highly readable list; the element example prompt statement includes possible element values ​​corresponding to the element type; the output format example prompt statement is used to provide an example of the output result format; Step S22, for the text position coordinate information in the text block list, constructing a text position coordinate information vector using a position coordinate projection layer; Step S23, inputting the investment transaction instruction image recognition prompt statement and the text position coordinate information vector together into the image transaction instruction parsing large model to generate a transaction instruction list consisting of multiple corresponding transaction instructions.

[0011] In conjunction with the first aspect, in one possible implementation, the large-scale image transaction instruction parsing model is an autoregressive pre-trained language model. The input to the large-scale image transaction instruction parsing model is a string of text and a vector of text position coordinates. The string of text includes a task description, task information, task requirements, and output format, used to guide and standardize the generation results of the large-scale transaction instruction parsing model. The vector of text position coordinates is used to guide the large-scale image transaction instruction parsing model in understanding the relative position of the text content on the original image. The construction steps of the large-scale image transaction instruction parsing model specifically include: Step S31, constructing a pre-trained alignment dataset for the position coordinate projection layer of the large-scale image transaction instruction parsing model. This pre-trained alignment dataset is constructed using data containing strong positional information associations, such as table documents and forms. For document images, etc., the text content and text location coordinates of the document images are used as input, and the structured information of the document images is used as labels to construct a supervised dataset; Step S32, construct a fine-tuning dataset for the large-scale image transaction instruction parsing model, using document question-and-answer data containing location-related information, taking the text content, text location coordinates of the document images, and document question-and-answer questions as input, and the answers to the document question-and-answer questions as labels to construct a supervised fine-tuning dataset; Step S33, use the image transaction instruction large-scale model pre-trained on the alignment dataset, and perform alignment training on the existing multimodal large-scale model to obtain the basic image transaction instruction large-scale model; Step S34, use the image transaction instruction recognition supervised fine-tuning dataset, and perform supervised fine-tuning training on the basic image transaction instruction large-scale model to obtain the image transaction instruction parsing large-scale model.

[0012] In conjunction with the first aspect, in one possible implementation, step S33 further includes: step S2321, establishing a large-scale dataset of table images and table structured representation text, wherein the syntax specification of the table structured representation text is HTML format, and performing text cleaning and filtering; step S2322, establishing a large-scale dataset of document images and document structured representation text, and using rules to remove low-quality data, wherein the syntax specification of the document structured representation text is Markdown format, and the syntax specification of the tables is HTML format.

[0013] In conjunction with the first aspect, in one possible implementation, step S3 specifically includes: step S31, traversing the transaction instruction list, using a business-related enterprise knowledge base to retrieve and match the elements of each transaction instruction, and supplementing the auxiliary elements based on the key elements; step S32, traversing the supplemented transaction instruction list, using a string comparison algorithm to compare the element values ​​with the entries in the business knowledge base, and correcting erroneous elements caused by errors in optical character recognition model recognition; step S33, traversing the supplemented transaction instruction list, disambiguating and repairing duplicate and conflicting elements, and obtaining the corrected and supplemented final transaction instruction list.

[0014] The second aspect of this application provides a multimodal large-scale model-based image financial instruction parsing system, which is used to implement any possible implementation of the first aspect to the first aspect of this application. The system includes: a multimodal large-scale language model pre-training module, which trains an existing multimodal large-scale language model using a pre-training alignment dataset of the image trading instruction parsing large-scale model's position coordinate projection layer to obtain a basic large-scale model of image trading instructions; a multimodal large-scale language model supervised fine-tuning module, which trains the basic large-scale model of image trading instructions using a fine-tuning dataset of the image trading instruction parsing large-scale model to obtain an image trading instruction parsing large-scale model; an instruction recognition module, which constructs a trading instruction recognition prompt statement from each independent instruction group text, inputs it into the image trading instruction parsing large-scale model, and generates a trading instruction list consisting of multiple corresponding trading instructions; a knowledge base completion and improvement module, which inputs the trading instruction list and uses an enterprise business data knowledge base to complete and correct the elements of the instructions, obtaining a final trading instruction list after correction and completion; and a continuous feedback learning module, which collects error examples encountered in business, analyzes the causes of errors using data points, generates an error example instruction fine-tuning dataset, and iteratively trains the model.

[0015] In conjunction with the second aspect, in one possible implementation, the large model pre-training module includes a multimodal large language model pre-training module and a multimodal large language model supervised fine-tuning module. The multimodal large language model pre-training module constructs table images and document images along with their corresponding structured text datasets based on the target use scenario of the large model parsing image transaction instructions, and performs module alignment training on the existing multimodal large language model. The large language model pre-training module trains on the multimodal large language model pre-training dataset through mask prediction learning; this pre-training method allows the model to learn the representation forms of tables and image documents. The multimodal large language model supervised fine-tuning module includes an investment instruction processing pipeline and a large language model supervised instruction fine-tuning module. The investment instruction processing pipeline cleans and filters the image investment instruction data collected from the business scenario, and manually... The system is structured as follows: A list of labeled investment instruction elements is assembled to generate an investment instruction fine-tuning dataset. The supervised fine-tuning module of the multimodal large language model is trained on this dataset through supervised learning. This involves converting the input investment instruction text into a list of investment instruction elements and learning to handle shared and nested elements. The instruction recognition module inputs independent investment instruction text into the image-based transaction instruction recognition large language model to obtain a list of investment instruction elements. The knowledge base completion module iterates through the list of investment instruction elements, processing each instruction object and completing the context for any elements that can be completed. The continuous feedback learning module collects erroneous recognition examples encountered by business personnel during system use, re-labels them, generates an investment instruction fine-tuning dataset of erroneous examples, and uses the large language model instruction fine-tuning module for instruction fine-tuning training to correct erroneous recognition issues.

[0016] A third aspect of this application provides a computer-readable storage medium comprising a computer program that, when run on an electronic device, causes the electronic device to execute the image financial instruction parsing method based on a multimodal large model as described in any possible implementation of the first aspect of this application.

[0017] The fourth aspect of this application provides an apparatus comprising a processor and a memory connected together, the processor being configured to execute a computer program stored in the memory to perform the image financial instruction parsing method based on a multimodal large model as described in any possible implementation of the first aspect of this application.

[0018] Compared with existing technologies, this invention demonstrates significant advantages in the innovative application of image investment transaction instruction recognition: By employing the technical solution of this invention, investment transaction instructions in images are precisely divided into independent instruction groups through an instruction group splitting language model. This innovative approach significantly improves the accuracy and processing speed of instruction recognition. Training with a large-scale image transaction instruction parsing model based on a multimodal large language model not only makes the process more convenient but also achieves unprecedented recognition accuracy. Particularly noteworthy is that the transaction instruction recognition prompts used in this invention can be flexibly applied across different transaction scenarios, effectively overcoming the limitations of existing technologies in application scenarios and opening up new paths for the efficient recognition and parsing of image investment transaction instructions.

[0019] This series of innovations not only enhances the intelligence level of image-based investment transaction instruction recognition, but also injects new vitality into the automated transaction process in the investment field, marking a crucial step forward for the investment industry in digital transformation and intelligent upgrading. Attached Figure Description

[0020] Figure 1 A flowchart illustrating a method for parsing image financial instructions based on a multimodal large model, provided in an embodiment of this application;

[0021] Figure 2 A flowchart illustrating a method for parsing image financial instructions based on a multimodal large model, provided in an embodiment of this application;

[0022] Figure 3 A flowchart of an optical character recognition process provided in this application embodiment;

[0023] Figure 4 A schematic diagram of the pre-training process for aligning a large model based on image transaction instructions is provided in this application embodiment;

[0024] Figure 5 This application provides a schematic diagram of a supervised fine-tuning training process for a large-scale image transaction instruction parsing model.

[0025] Figure 6 A reasoning flowchart for a large-scale model based on image transaction instruction parsing is provided in this application embodiment;

[0026] Figure 7 This is a schematic diagram of an image-based financial instruction parsing system based on a multimodal large model, provided as an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] The term "and / or" appearing in this application can describe the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this application generally indicates that the preceding and following related objects have an "or" relationship.

[0029] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] This application provides a method, system, computer-readable storage medium, and device for parsing financial instructions from images based on a multimodal large-scale model. The method, through deep learning technology, can accurately extract trading instruction information from images, maintaining high recognition accuracy even in complex backgrounds or with blurred text. The system utilizes the pre-training capabilities of a multimodal large-scale language model, combined with natural language processing and image recognition technologies. Through alignment pre-training and supervised fine-tuning, it enhances the understanding of investment-related terminology and symbols, ensuring the accuracy of instruction parsing.

[0031] For details, please see Figure 1 This application provides a method for parsing financial instructions from images based on a multimodal large model, the method comprising:

[0032] S1. Input the original document image containing investment trading instructions, and use the optical character recognition model to perform text recognition on the original document image to obtain a list of text blocks containing text content and text coordinates.

[0033] S2. For the list of text blocks, construct image transaction instruction parsing prompts and location information vectors, and input the image transaction instruction parsing prompts and location information vectors into the image transaction instruction parsing model. The image transaction instruction parsing model outputs the corresponding transaction instruction list, which contains all the transaction instructions in the image.

[0034] S3. For the list of trading instructions, each instruction is compared and verified against the trading business knowledge base. Additional elements are added according to business rules, and text errors caused by optical character recognition errors are corrected according to text features to ensure its completeness and accuracy.

[0035] S4. Output the complete transaction instruction after the above processing steps.

[0036] Furthermore, step S2 includes:

[0037] Step S21: For the text content in the text block list, construct an investment transaction instruction image recognition prompt statement. The investment transaction instruction image recognition prompt statement includes one or more of the following: A. Task description prompt statement, B. Target recognition element name list prompt statement, C. Element example prompt statement, D. Output format example prompt statement, E. Investment transaction instruction image text content to be parsed.

[0038] The task description prompt statement is used to explain the text source and output format of the task target; the target identification element name list prompt statement is used to display the element names in a highly readable list; the element example prompt statement contains the possible element values ​​corresponding to the element type; and the output format example prompt statement is used to provide an example of the output result format.

[0039] Step S22: For the text position coordinate information in the text block list, construct the text position coordinate information vector using the position coordinate projection layer;

[0040] Step S23: Input the image recognition prompt of the investment transaction instruction and the text position coordinate information vector into the image transaction instruction parsing model to generate a transaction instruction list consisting of multiple corresponding transaction instructions.

[0041] The image transaction instruction parsing model is an autoregressive pre-trained language model. Its input consists of a string of text and a vector of text position coordinates. The string of text includes a task description, task information, task requirements, and output format, used to guide and standardize the model's output. The text position coordinate vector guides the model in understanding the relative position of the text content within the original image. The specific steps for constructing the image transaction instruction parsing model include:

[0042] Step S31: Construct a pre-trained alignment dataset for the position coordinate projection layer of the large model for parsing image transaction instructions. The pre-trained alignment dataset is constructed using data containing strong positional information associations. Table document images, form document images, etc. are used as inputs, and the structured information of the document images is used as labels to construct a supervised dataset.

[0043] Step S32: Construct a large-scale model fine-tuning dataset for image transaction instruction parsing. Using document question-and-answer data containing location-related information, the text content of the document image, the text location coordinates, and the document question-and-answer questions are used as inputs, and the answers to the document question-and-answer questions are used as labels to construct a supervised fine-tuning dataset.

[0044] Step S33: Use the image transaction instruction large model to pre-train the alignment dataset, and perform alignment training on the basis of the existing multimodal large model to obtain the image transaction instruction basic large model.

[0045] Step S34: Use the image transaction instruction identification supervised fine-tuning dataset, perform supervised fine-tuning training on the basic large model of image transaction instructions, and obtain the large model of image transaction instruction parsing.

[0046] Furthermore, step S33 also includes:

[0047] Step S2321: Establish a large-scale dataset of table images and table structured representation text, wherein the syntax of the table structured representation text is HTML format, and perform text cleaning and filtering.

[0048] Step S2322: Establish a large-scale dataset of document images and document structured representation text, and use rules to remove low-quality data. The syntax of the document structured representation text is Markdown format, and the syntax of the tables is HTML format.

[0049] It should be noted that step S3 specifically includes:

[0050] Step S31: Traverse the list of transaction instructions, use the business-related enterprise knowledge base to search and match the elements of each transaction instruction, and complete the auxiliary elements based on the key elements;

[0051] Step S32: Traverse the completed transaction instruction list, use a string comparison algorithm to compare the element values ​​with the entries in the business knowledge base, and correct erroneous elements caused by the optical character recognition model's recognition error.

[0052] Step S33: Traverse the completed transaction instruction list, disambiguate and repair duplicate and conflicting elements, and obtain the final transaction instruction list after correction and completion.

[0053] like Figure 2 This application also provides a method for parsing document images containing investment trading instructions into a list of investment trading instructions based on a multimodal large language model:

[0054] Step 1000: Use an optical character recognition model to identify characters in the image:

[0055] Text recognition is performed on images containing investment transaction instructions using an optical character recognition model.

[0056] Specifically, step 1000 includes steps 1100-1200, please refer to [link / reference]. Figure 3 :

[0057] Step 1100: Image of the document containing the trading instructions.

[0058] An image containing investment trading instructions.

[0059] Step 1200: Optical Character Recognition Model (OCR Model).

[0060] The image containing the investment transaction instructions obtained in step 1100 is input into the optical character recognition model (OCR model).

[0061] Step 1300: Text information and text coordinate information.

[0062] The text information and text location coordinate information generated in step 1200 are organized and normalized.

[0063] Step 2000: Train the basic large model of image trading orders, and train the large model of image trading order parsing.

[0064] The training of a large-scale image trading instruction parsing model will employ a combination of alignment pre-training and instruction fine-tuning training. First, the existing multimodal large-scale model will be pre-trained using formatted text containing tables and documents to obtain a basic large-scale image trading instruction model. Then, the basic large-scale image trading instruction model will be subjected to supervised fine-tuning training using an instruction fine-tuning dataset consisting of images containing investment trading instructions and corresponding instruction annotations to obtain a large-scale trading instruction parsing model.

[0065] For details, please see Figure 4 Step 2000 involves training the basic large model of image trading instructions, which includes steps 2110-2160.

[0066] Step 2110: Table and document data.

[0067] Images of tables and documents are obtained from publicly available table and document datasets. Their corresponding formatted text annotations are then converted to the specified format of the aligned pre-training dataset. The aligned pre-training dataset is used to perform alignment pre-training on an existing multimodal large-scale model to obtain a basic large-scale model for image transaction instructions. Alternatively, images of tables and documents along with their corresponding formatted text annotations are obtained from publicly available table and document datasets. If no corresponding annotations are found, a document format parsing tool is used to parse the format and obtain the corresponding formatted text.

[0068] Step 2120: Convert to a text representation in the specified format.

[0069] Organize the tabular and document data obtained in step 2110 into a unified formatted text structure.

[0070] Step 2130: Fine-tune the instruction prompt template.

[0071] Depending on the type of training sample, corresponding training prompt text templates are created for table images and document images respectively.

[0072] Step 2140: Align the dataset with the multimodal big data model.

[0073] The images, their corresponding formatted text, and the corresponding training prompt text templates are assembled and processed into a multimodal large language model aligned pre-training dataset.

[0074] Step 2150: Large-scale pre-training of multimodal language models.

[0075] Obtain existing large-scale pre-trained multimodal language models.

[0076] Step 2160: Basic Model of Image Trading Instructions

[0077] The multimodal large language model obtained in step 2150 is used to train on the pre-training dataset aligned with the multimodal large language model obtained in step 2140 to obtain the basic large model of image transaction instructions.

[0078] Please see Figure 5 The training of the large model for parsing image trading instructions includes steps 2210-2290.

[0079] Step 2210: Image investment transaction instruction data.

[0080] Organize and filter image transaction instruction data from business data.

[0081] Step 2220: Label feature information.

[0082] In the image transaction instruction data obtained in step 2210, transaction instruction elements are labeled, which indicates the instruction combination contained in the transaction instruction image and the information such as the element type and element value in the combination.

[0083] Step 2230: List of investment transaction instructions.

[0084] By combining instructions with elements, the target output of the model, a list of investment and trading instructions, is generated.

[0085] Step 2240: Serialization tool.

[0086] Use a JSON list serialization tool to serialize the list of investment transaction instructions obtained in step 2230.

[0087] Step 2250: Instruction fine-tuning prompt template.

[0088] Write corresponding prompt text templates based on the business scenario and the target output of the business.

[0089] Step 2260: Supervised fine-tuning of the dataset for a multimodal large language model.

[0090] The images, text and location coordinates obtained from the images through an optical character recognition model, a serialized list of investment and trading instructions, and prompt text templates are combined to generate a supervised fine-tuning dataset for a multimodal large language model.

[0091] Step 2270: Basic Model of Image Trading Instructions

[0092] Obtain the aligned pre-trained image trading instruction base model from step 2160.

[0093] Step 2280: Low-rank adaptation layer fine-tuning method.

[0094] The low-rank adaptation layer fine-tuning method is used to perform supervised fine-tuning on the image transaction instruction base model obtained in step 2270 and the multimodal language model obtained in step 2260 on the supervised fine-tuning dataset.

[0095] Step 2290: Image transaction instruction parsing large model.

[0096] Obtain the trained image trading instruction parsing large model from step 2280.

[0097] Step 3000: Recognize transaction instructions from images and text.

[0098] After training the large-scale image trading instruction parsing model, the image containing the trading instruction to be recognized, along with the text and location information recognized by the optical character recognition model, are input into the large-scale image trading instruction parsing model for trading instruction recognition.

[0099] For details, please see Figure 6 This step 3000 specifically includes:

[0100] Step 3100: Image of the document containing the transaction instructions.

[0101] Images containing trading instructions need to be resized from their original size to a fixed size, and then the images need to be normalized to obtain the trading instruction images to be identified.

[0102] Step 3200: Text information and text location coordinate information.

[0103] The text and location information obtained from the original transaction instruction image through an optical character recognition model are matched and normalized.

[0104] Step 3300: Task prompt.

[0105] Select the appropriate task prompt text based on the business scenario, and specify in the prompt text the type of elements that the large model of image transaction instruction parsing should output.

[0106] Step 3400: Image transaction instruction parsing large model.

[0107] Input the transaction instruction images, texts, location information, and task prompt text obtained in steps 3100, 3200, and 3300 into the image transaction instruction parsing model.

[0108] Step 3500: List of trading instructions.

[0109] The transaction instruction list text is obtained from step 3400. Then, the corresponding deserialization tool is used to deserialize the text to obtain the transaction instruction list.

[0110] Step 4000: Use the knowledge base for retrieval and matching.

[0111] Based on the elements involved in the business, the corresponding business information database is invoked to retrieve and match each instruction object in the instruction list. Based on the information of some instruction elements, the information of other missing elements is supplemented. For example, by querying the database through the bond code element that has already appeared, the bond name corresponding to the bond code is obtained, and the missing element value is filled with this bond name.

[0112] Finally, after completion and refinement, the output is the final content of this solution and system.

[0113] This invention also provides an image-based financial instruction parsing system based on a multimodal large model, which is based on the aforementioned image-based financial instruction parsing method based on a multimodal large model. The system includes:

[0114] The multimodal large language model pre-training module uses the image transaction instruction parsing large model position coordinate projection layer pre-training alignment dataset to train the existing multimodal large language model, thus obtaining the basic large model of image transaction instructions;

[0115] The multimodal large language model has a supervised fine-tuning module. The basic large model of image transaction instructions is trained using the image transaction instruction parsing large model fine-tuning dataset to obtain the image transaction instruction parsing large model.

[0116] The instruction recognition module constructs a transaction instruction recognition prompt statement from each independent instruction group text, and generates a transaction instruction list consisting of multiple corresponding transaction instructions from the input image transaction instruction parsing large model;

[0117] The knowledge base completion and improvement module takes a list of transaction instructions as input and uses the enterprise business data knowledge base to complete and correct the elements of the instructions, resulting in a final list of transaction instructions after correction and completion.

[0118] The continuous feedback learning module collects error examples encountered in business operations, analyzes the causes of errors through data entry points, generates error example instruction fine-tuning datasets, and iteratively trains the model.

[0119] The large model pre-training module includes a multimodal large language model pre-training module and a multimodal large language model supervised fine-tuning module;

[0120] The multimodal large language model pre-training module parses the target use scenario of the large model based on the image transaction instructions, constructs table images and document images and their corresponding structured text datasets, and performs module alignment training on the existing multimodal large language model.

[0121] The large language model pre-training module is trained on a multimodal large language model pre-training dataset through mask prediction learning. This pre-training method enables the model to learn the representation of tables and image documents.

[0122] The multimodal large language model supervised fine-tuning module includes an investment instruction processing pipeline and a large language model supervised instruction fine-tuning module;

[0123] The investment instruction processing pipeline cleans and filters the image investment instruction data collected in the business scenario, manually annotates the list of investment instruction elements, and assembles and generates an investment instruction fine-tuning dataset.

[0124] The supervised fine-tuning module of the multimodal large language model is trained on the investment instruction fine-tuning dataset through supervised learning. It needs to convert the input investment instruction corpus text into a list of investment instruction elements and learn to process shared elements and nested elements.

[0125] The instruction recognition module inputs the independent investment instruction corpus into the image transaction instruction recognition language model to obtain a list of investment instruction element sets.

[0126] The knowledge base completion and improvement module will traverse the list of investment instruction elements one by one, process each instruction object in the list, and complete the context of the elements that can be completed.

[0127] The continuous feedback learning module collects error recognition examples encountered by business personnel during system use, re-labels them, generates an investment instruction fine-tuning dataset of error examples, and uses the large language model instruction fine-tuning module to perform instruction fine-tuning training to fix error recognition problems.

[0128] Embodiments of the present invention also disclose a computer-readable storage medium comprising a computer program that, when run on an electronic device, causes the electronic device to execute the image financial instruction parsing method based on a multimodal large model as described above.

[0129] Embodiments of the present invention also disclose an apparatus comprising a processor and a memory connected together, the processor being configured to execute a computer program stored in the memory to perform the image financial instruction parsing method based on a multimodal large model as described above.

[0130] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

[0131] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0132] In the embodiments provided in this application, it should be understood that the disclosed methods can be implemented in other ways without departing from the spirit and scope of this application. The current embodiments are merely exemplary examples and should not be considered limiting, nor should the specific content given limit the purpose of this application. For example, some features may be omitted or not implemented.

[0133] The technical means disclosed in this application are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

[0134] The foregoing has provided a detailed description of a method and related apparatus for parsing image financial instructions based on a multimodal large model, as provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for parsing financial instructions from images based on a multimodal large model, characterized in that, The method includes: Step S1: Input the original document image containing investment transaction instructions, and use an optical character recognition model to perform text recognition on the original document image to obtain a list of text blocks containing text content and text coordinates; Step S2: For the text block list, construct image transaction instruction parsing prompt statements and location information vectors, and input the image transaction instruction parsing prompt statements and location information vectors into the image transaction instruction parsing big model. The image transaction instruction parsing big model outputs the corresponding transaction instruction list, which contains all the transaction instructions in the image. Step S3: For the transaction instruction list, each instruction is compared and verified with the transaction business knowledge base, supplementary elements are added according to business rules, and text errors caused by optical character recognition errors are corrected according to text features to ensure its completeness and accuracy. Step S4: Output the complete transaction instruction after the above processing steps.

2. The image financial instruction parsing method based on a multimodal large model according to claim 1, characterized in that, Step S2 includes: Step S21: For the text content in the text block list, construct an investment transaction instruction image recognition prompt statement. The investment transaction instruction image recognition prompt statement includes one or more of the following: A. Task description prompt statement, B. Target recognition element name list prompt statement, C. Element example prompt statement, D. Output format example prompt statement, E. Investment transaction instruction image text content to be parsed. The task description prompt statement is used to explain the text source and output format of the task target; the target identification element name list prompt statement is used to display the element names in a highly readable list; the element example prompt statement contains the possible element values ​​corresponding to the element type; and the output format example prompt statement is used to provide an example of the output result format. Step S22: For the text position coordinate information in the text block list, construct the text position coordinate information vector using the position coordinate projection layer; Step S23: Input the image recognition prompt of the investment transaction instruction and the text position coordinate information vector into the image transaction instruction parsing model to generate a transaction instruction list consisting of multiple corresponding transaction instructions.

3. The image financial instruction parsing method based on a multimodal large model according to claim 2, characterized in that, The image transaction instruction parsing model is an autoregressive pre-trained language model. Its input consists of a string of text and a vector of text position coordinates. The string of text includes a task description, task information, task requirements, and output format, used to guide and standardize the model's output. The text position coordinate vector guides the model in understanding the relative position of the text content within the original image. The specific steps for constructing the image transaction instruction parsing model include: Step S31: Construct a pre-trained alignment dataset for the position coordinate projection layer of the large model for parsing image transaction instructions. The pre-trained alignment dataset is constructed using data containing strong positional information associations. Table document images, form document images, etc. are used as inputs, and the structured information of the document images is used as labels to construct a supervised dataset. Step S32: Construct a large-scale model fine-tuning dataset for image transaction instruction parsing. Using document question-and-answer data containing location-related information, the text content of the document image, the text location coordinates, and the document question-and-answer questions are used as inputs, and the answers to the document question-and-answer questions are used as labels to construct a supervised fine-tuning dataset. Step S33: Use the image transaction instruction large model to pre-train the alignment dataset, and perform alignment training on the basis of the existing multimodal large model to obtain the image transaction instruction basic large model. Step S34: Use the image transaction instruction identification supervised fine-tuning dataset, perform supervised fine-tuning training on the basic large model of image transaction instructions, and obtain the large model of image transaction instruction parsing.

4. The image financial instruction parsing method based on a multimodal large model according to claim 3, characterized in that, Step S33 also includes: Step S2321: Establish a large-scale dataset of table images and table structured representation text, wherein the syntax of the table structured representation text is HTML format, and perform text cleaning and filtering. Step S2322: Establish a large-scale dataset of document images and document structured representation text, and use rules to remove low-quality data. The syntax of the document structured representation text is Markdown format, and the syntax of the tables is HTML format.

5. The image financial instruction parsing method based on a multimodal large model according to claim 1, characterized in that, Step S3 specifically includes: Step S31: Traverse the list of transaction instructions, use the business-related enterprise knowledge base to search and match the elements of each transaction instruction, and complete the auxiliary elements based on the key elements; Step S32: Traverse the completed transaction instruction list, use a string comparison algorithm to compare the element values ​​with the entries in the business knowledge base, and correct erroneous elements caused by the optical character recognition model's recognition error. Step S33: Traverse the completed transaction instruction list, disambiguate and repair duplicate and conflicting elements, and obtain the final transaction instruction list after correction and completion.

6. A picture-based financial instruction parsing system based on a multimodal large model, characterized in that, The system is applied to the image financial instruction parsing method based on a multimodal large model as described in any one of claims 1-5, and the system comprises: The multimodal large language model pre-training module uses the image transaction instruction parsing large model position coordinate projection layer pre-training alignment dataset to train the existing multimodal large language model, thus obtaining the basic large model of image transaction instructions; The multimodal large language model has a supervised fine-tuning module. The basic large model of image transaction instructions is trained using the image transaction instruction parsing large model fine-tuning dataset to obtain the image transaction instruction parsing large model. The instruction recognition module constructs a transaction instruction recognition prompt statement from each independent instruction group text, and generates a transaction instruction list consisting of multiple corresponding transaction instructions from the input image transaction instruction parsing large model; The knowledge base completion and improvement module takes a list of transaction instructions as input and uses the enterprise business data knowledge base to complete and correct the elements of the instructions, resulting in a final list of transaction instructions after correction and completion. The continuous feedback learning module collects error examples encountered in business operations, analyzes the causes of errors through data entry points, generates error example instruction fine-tuning datasets, and iteratively trains the model.

7. The image financial instruction parsing system based on a multimodal large model according to claim 6, characterized in that, The large model pre-training module includes a multimodal large language model pre-training module and a multimodal large language model supervised fine-tuning module; The multimodal large language model pre-training module parses the target use scenario of the large model based on the image transaction instructions, constructs table images and document images and their corresponding structured text datasets, and performs module alignment training on the existing multimodal large language model. The large language model pre-training module is trained on a multimodal large language model pre-training dataset through mask prediction learning. This pre-training method enables the model to learn the representation of tables and image documents. The multimodal large language model supervised fine-tuning module includes an investment instruction processing pipeline and a large language model supervised instruction fine-tuning module; The investment instruction processing pipeline cleans and filters the image investment instruction data collected in the business scenario, manually annotates the list of investment instruction elements, and assembles and generates an investment instruction fine-tuning dataset. The supervised fine-tuning module of the multimodal large language model is trained on the investment instruction fine-tuning dataset through supervised learning. It needs to convert the input investment instruction corpus text into a list of investment instruction elements and learn to process shared elements and nested elements. The instruction recognition module inputs the independent investment instruction corpus into the image transaction instruction parsing model to obtain a list of investment instruction element sets. The knowledge base completion and improvement module will traverse the list of investment instruction elements one by one, process each instruction object in the list, and complete the context of the elements that can be completed. The continuous feedback learning module collects error recognition examples encountered by business personnel during system use, re-labels them, generates an investment instruction fine-tuning dataset of error examples, and uses the large language model instruction fine-tuning module to perform instruction fine-tuning training to fix error recognition problems.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program that, when run on an electronic device, causes the electronic device to execute the image financial instruction parsing method based on a multimodal large model as described in any one of claims 1 to 5.

9. An apparatus comprising a processor and a memory connected together, characterized in that: The processor is used to execute the computer program stored in the memory to perform the image financial instruction parsing method based on a multimodal large model as described in any one of claims 1 to 5.

Images