Picture financial instruction analysis method based on multi-modal large model and related device

Abstract

The invention provides a picture financial instruction analysis method based on a multi-modal large model and a related device. The method comprises the following steps: S1, inputting an original document picture containing an investment transaction instruction, and performing text recognition on the original document picture by using an optical character recognition model to obtain a text block list; s2, aiming at the text block list, constructing a picture transaction instruction analysis prompt statement and a position information vector, inputting the picture transaction instruction analysis prompt statement and the position information vector into a picture transaction instruction analysis large model, and outputting a corresponding transaction instruction list by the picture transaction instruction analysis large model, the transaction instruction list comprises all transaction instructions in the picture; s3, aiming at the transaction instruction list, comparing and verifying the transaction instruction list with the transaction business knowledge base, supplementing auxiliary elements according to business rules, and correcting text error elements caused by optical character recognition errors according to text features; and S4, outputting the complete transaction instruction after the processing steps.

Description

Technical Field

[0001] The present invention relates to the field of picture text element recognition, and specifically to a picture financial instruction parsing method and related device based on a multimodal large model. Background Art

[0002] In the digital transformation of the investment field, the recognition and analysis of image trading instructions has become a key link in improving trading efficiency. Traditionally, the instructions conveyed by trading managers to traders often appear in the form of pictures, which contain rich trading information, including key data such as stock codes, trading volumes, and prices.

[0003] However, image instructions are often accompanied by complex backgrounds, blurred handwriting, or industry-specific symbols and terms, which poses considerable challenges to the accurate understanding and rapid execution of instructions. Summary of the invention

[0004] To address this problem, this application provides a method, system, computer-readable storage medium and device for parsing financial instructions from images based on a multimodal large model. This method uses deep learning technology to accurately extract transaction instruction information from images, and can maintain a high recognition accuracy rate even in complex backgrounds or blurred handwriting. The system uses the pre-training capabilities of a multimodal large language model, combined with natural language processing and image recognition technology, to enhance the understanding of professional terms and symbols in the investment field through alignment pre-training and supervised fine-tuning, ensuring the accuracy of instruction parsing.

[0005] The application of this innovative method has brought significant changes to the investment field. First, the automatic image instruction parsing technology can significantly reduce the labor costs of fund companies, shorten the transaction cycle, and significantly improve work efficiency. Secondly, through the powerful parameter advantages of the large language model, the system can more accurately understand the information in the image instructions, reduce dependence on manual intervention, and improve the accuracy and efficiency of transactions. Finally, through the linkage with the enterprise knowledge base, the system can further optimize the post-processing of image instructions, realize rule verification and matching in business logic, and ensure the compliance and security of transactions.

[0006] In summary, the image trading instruction recognition technology based on a multimodal large language model not only represents the latest progress in automation technology in the investment field, but also provides strong technical support for improving transaction efficiency, reducing costs, and ensuring transaction accuracy, marking an important milestone in the intelligent transformation of the investment field.

[0007] The present invention aims to overcome the efficiency and accuracy problems faced by the traditional manual analysis of image trading instructions. In the traditional analysis process, relying on manual reading and understanding of image instructions not only consumes a lot of time and human resources, but is also easily affected by subjective judgment, increasing the potential risk of analysis errors. Faced with the growing trading volume and complex and changing market environment, the investment field urgently needs a faster and smarter solution to significantly improve the efficiency and accuracy of instruction recognition.

[0008] The image transaction instruction parsing method and system based on the multimodal large language model proposed in the present invention can realize the automation and intelligence of instruction parsing, significantly speed up the parsing speed, greatly reduce the dependence on manpower, and thus significantly improve work efficiency. The introduction of the large language model can not only process image instructions under complex backgrounds, but also understand industry-specific symbols and terms through deep learning, reduce parsing errors, and ensure the accurate execution of transaction instructions. This innovative solution marks a solid step forward in the automation and intelligence of the investment field, and provides strong technical support for improving transaction efficiency, reducing operating costs, and enhancing market competitiveness.

[0009] In view of this, the first aspect of the present application provides a method for parsing image financial instructions based on a multimodal large model. The method includes: step S1, inputting an original document image containing investment transaction instructions, using an optical character recognition model to perform text recognition on the original document image, and obtaining a text block list containing text content and text coordinates; step S2, for the text block list, constructing an image transaction instruction parsing prompt statement and a position information vector, and inputting the image transaction instruction parsing prompt statement and the position information vector into the image transaction instruction parsing large model, and the image transaction instruction parsing large model outputs a corresponding transaction instruction list, and the transaction instruction list contains all transaction instructions in the image; step S3, for the transaction instruction list, each instruction is compared and verified with the transaction business knowledge base, supplemented with auxiliary elements according to business rules, and corrected according to text features. The text error elements caused by optical character recognition errors to ensure its completeness and accuracy; step S4, outputting the complete transaction instruction after the above processing steps.

[0010] In combination with the first aspect, in a possible implementation method, the step S2 includes: step S21, for the text content in the text block list, construct an investment transaction instruction image recognition prompt statement, the investment transaction instruction image recognition prompt statement includes one or more of the following: A task description prompt statement, B target identification element name list prompt statement, C element example prompt statement, D output format sample prompt statement, E investment transaction instruction image text content to be parsed; wherein, the task description prompt statement is used to explain the text source and output format of the identification task target; the target identification element name list prompt statement is used to display the element name in a highly readable list; the element example prompt statement contains possible element values ​​corresponding to the element type; the output format sample prompt statement is used to give an output result format example; step S22, for the text position coordinate information in the text block list, use the position coordinate projection layer to construct a text position coordinate information vector; step S23, input the investment transaction instruction image recognition prompt statement and the text position coordinate information vector together into the image transaction instruction parsing large model to generate a corresponding transaction instruction list consisting of multiple transaction instructions.

[0011] In combination with the first aspect, in a possible implementation method, the image transaction instruction parsing large model is an autoregressive pre-trained language model, and the input of the image transaction instruction parsing large model is a string text and a text position coordinate information vector, the string text includes a task description, task information, task requirements, and an output format, which are used to guide and standardize the generation results of the transaction instruction parsing large model; the text position coordinate information vector is used to guide the image transaction instruction parsing large model to understand the relative position of the text content on the original image, and the construction steps of the image transaction instruction parsing large model specifically include: step S31, constructing a pre-trained alignment data set for the position coordinate projection layer of the image transaction instruction parsing large model, using data containing strong position information association to construct a pre-trained alignment data set, using table document images, form Document images, etc., take the text content and text position coordinate information of the document image as input, and the structured information of the document image as a label to construct a supervised data set; step S32, construct a fine-tuning data set for the image transaction instruction parsing model, use the document question and answer data containing location-related information, take the text content of the document image, text position coordinate information and the questions of the document question and answer as input, and the answers to the document question and answer as labels to construct a supervised fine-tuning data set; step S33, use the image transaction instruction model pre-training alignment data set, perform alignment training on the basis of the existing multimodal large model, and obtain the image transaction instruction basic large model; step S34, use the image transaction instruction recognition supervised fine-tuning data set, perform supervised fine-tuning training on the image transaction instruction basic large model, and obtain the image transaction instruction parsing large model.

[0012] In combination with the first aspect, in a possible implementation method, step S33 also includes: step S2321, establishing a data set of large-scale table images and table structured representation texts, wherein the grammatical specification of the table structured representation texts is in HTML format, and performing text cleaning and filtering; step S2322, establishing a data set of large-scale document images and document structured representation texts, and using rules to eliminate low-quality data, wherein the grammatical specification of the document structured representation texts is in Markdown format, and the grammatical specification of the table is in HTML format.

[0013] In combination with the first aspect, in a possible implementation method, the step S3 specifically includes: step S31, traversing the transaction instruction list, using the business-related enterprise knowledge base to search and match the elements of each transaction instruction, and completing the subsidiary elements according to the key elements; step S32, traversing the completed transaction instruction list, using a string comparison algorithm, comparing the element value with the entry in the business knowledge base, and correcting the erroneous elements caused by the recognition error of the optical character recognition model; step S33, traversing the completed transaction instruction list, disambiguating and repairing the repeated conflicting elements, and obtaining the final transaction instruction list after correction and completion.

[0014] The second aspect of the present application provides a picture financial instruction parsing system based on a multimodal large model, which is used to implement any possible implementation method of the first aspect to the first aspect of the present application. The system includes: a multimodal large language model pre-training module, which uses the image transaction instruction parsing large model position coordinate projection layer pre-training alignment data set to train the existing multimodal large language model to obtain the basic large model of the image transaction instruction; a multimodal large language model supervised fine-tuning module, which uses the image transaction instruction parsing large model fine-tuning data set to train the basic large model of the image transaction instruction to obtain the image transaction instruction parsing large model; an instruction recognition module, which constructs each independent instruction group text into a transaction instruction recognition prompt statement, inputs the image transaction instruction parsing large model, and generates a corresponding transaction instruction list consisting of multiple transaction instructions; a knowledge base completion module, which inputs the transaction instruction list, uses the enterprise business data knowledge base, completes and corrects the elements of the instruction, and obtains the final transaction instruction list after correction and completion; a continuous feedback learning module, which collects error samples encountered in the business, analyzes the cause of the error, generates an error sample instruction fine-tuning data set, and iteratively trains the model.

[0015] In combination with the second aspect, in a possible implementation, the large model pre-training module includes a multimodal large language model pre-training module and a multimodal large language model supervised fine-tuning module; wherein the multimodal large language model pre-training module, according to the target usage scenario of the image transaction instruction parsing large model, constructs table images and document images and their corresponding structured text data sets, and performs module alignment training on the existing multimodal large language model; the large language model pre-training module is trained on the multimodal large language model pre-training data set through mask prediction learning. This pre-training method enables the model to learn the representation of tables and image documents; the multimodal large language model supervised fine-tuning module includes an investment instruction processing pipeline and a large language model supervised instruction fine-tuning module; wherein the investment instruction processing pipeline cleans and filters the image investment instruction data collected in the business scenario, and manually Label the list of investment instruction element sets and assemble to generate an investment instruction fine-tuning dataset; the multimodal large language model supervised fine-tuning module is trained on the investment instruction fine-tuning dataset through supervised learning, and needs to convert the input investment instruction corpus text recognition into an investment instruction element list, learn to process shared elements and nested elements; the instruction recognition module inputs the independent investment instruction corpus into the image transaction instruction recognition large language model to obtain the investment instruction element set list; the knowledge base completion module will go through the investment instruction element set list one by one, process each instruction object in the list, and complete the completeable elements in the context; the continuous feedback learning module collects the error recognition samples encountered by business personnel during the use of the system, re-labels them, generates an investment instruction fine-tuning dataset of error samples, uses the large language model instruction fine-tuning module to perform instruction fine-tuning training, and fixes the error recognition problem.

[0016] The third aspect of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program runs on an electronic device, the electronic device executes the image financial instruction parsing method based on a multimodal large model as described in any possible implementation method of the first aspect to the first aspect of the present application.

[0017] The fourth aspect of the present application provides a device comprising a connected processor and a memory, wherein the processor is used to execute a computer program stored in the memory to execute the image financial instruction parsing method based on a multimodal large model as described in any possible implementation method of the first aspect to the first aspect of the present application.

[0018] Compared with the prior art, the innovative application of the present invention in the field of image investment and trading instruction recognition shows significant advantages: adopting the technical solution of the present invention, the investment and trading instructions in the image are accurately divided into independent instruction groups through the instruction group splitting language model. This innovative measure significantly improves the accuracy and processing speed of instruction recognition. The use of a large model for parsing image trading instructions based on a multimodal large language model for training not only makes the process more convenient, but also achieves an unprecedented level of recognition accuracy. It is particularly worth mentioning that the transaction instruction recognition prompt statement adopted by the present invention can be flexibly migrated and applied in different transaction scenarios, effectively breaking through the limitations of the prior art in application scenarios, and opening up a new path for the efficient recognition and parsing of image investment and trading instructions.

[0019] This series of innovations not only improves the intelligence level of image investment transaction instruction recognition, but also injects new vitality into the automated trading process in the investment field, marking a key step in the investment industry's digital transformation and intelligent upgrading. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a method for parsing financial instructions from images based on a multimodal large model provided in an embodiment of the present application;

[0021] Figure 2 A flowchart of a method for parsing financial instructions from images based on a multimodal large model provided in an embodiment of the present application;

[0022] Figure 3 An optical character recognition flow chart provided for an embodiment of the present application;

[0023] Figure 4 A schematic diagram of a pre-training process for aligning a basic large model of an image transaction instruction provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of a supervised fine-tuning training process of a large model for parsing image transaction instructions provided in an embodiment of the present application;

[0025] Figure 6 A reasoning flow chart based on a large model for parsing image transaction instructions provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of a picture financial instruction parsing system based on a multimodal large model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0028] The term "and / or" in this application can be a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the associated objects before and after are in an "or" relationship.

[0029] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] The present application provides a method, system, computer-readable storage medium and device for parsing financial instructions from images based on a multimodal large model. The method uses deep learning technology to accurately extract transaction instruction information from images, and can maintain a high recognition accuracy rate even in complex backgrounds or blurred handwriting. The system uses the pre-training capability of a multimodal large language model, combined with natural language processing and image recognition technology, to enhance the understanding of professional terms and symbols in the investment field through alignment pre-training and supervised fine-tuning, ensuring the accuracy of instruction parsing.

[0031] For details, see Figure 1 The present application provides a method for parsing picture financial instructions based on a multimodal large model, the method comprising:

[0032] S1. Input an original document image containing investment transaction instructions, use an optical character recognition model to perform text recognition on the original document image, and obtain a text block list containing text content and text coordinates.

[0033] S2. For the text block list, construct a picture transaction instruction parsing prompt sentence and a position information vector, and input the picture transaction instruction parsing prompt sentence and the position information vector into the picture transaction instruction parsing large model, and the picture transaction instruction parsing large model outputs the corresponding transaction instruction list, which contains all the transaction instructions in the picture.

[0034] S3. For the transaction instruction list, each instruction is compared and verified with the transaction business knowledge base, supplemented with additional elements according to business rules, and corrected according to text features. Errors in the text caused by optical character recognition errors ensure its completeness and accuracy.

[0035] S4. Output the complete transaction instruction after the above processing steps.

[0036] Furthermore, step S2 includes:

[0037] Step S21, constructing an investment transaction instruction picture recognition prompt sentence for the text content in the text block list, wherein the investment transaction instruction picture recognition prompt sentence includes one or more of the following: A task description prompt sentence, B target recognition element name list prompt sentence, C element example prompt sentence, D output format sample prompt sentence, and E investment transaction instruction picture text content to be parsed;

[0038] Among them, the task description prompt statement is used to explain the text source and output format of the recognition task target; the target recognition element name list prompt statement is used to display the element name in a highly readable list; the element example prompt statement contains the possible element values ​​corresponding to the element type; the output format sample prompt statement is used to give an example of the output result format;

[0039] Step S22, for the text position coordinate information in the text block list, using the position coordinate projection layer to construct a text position coordinate information vector;

[0040] Step S23, inputting the investment transaction instruction picture recognition prompt sentence and the text position coordinate information vector into the picture transaction instruction parsing model together to generate a transaction instruction list consisting of a corresponding plurality of transaction instructions.

[0041] The image transaction instruction parsing model is an autoregressive pre-trained language model. The input of the image transaction instruction parsing model is a string text and a text position coordinate information vector. The string text includes a task description, task information, task requirements, and output format, which are used to guide and standardize the generation result of the transaction instruction parsing model; the text position coordinate information vector is used to guide the image transaction instruction parsing model to understand the relative position of the text content on the original image. The steps of constructing the image transaction instruction parsing model specifically include:

[0042] Step S31, constructing a pre-trained alignment dataset for the position coordinate projection layer of the large model for parsing image transaction instructions, using data containing strong position information association to construct the pre-trained alignment dataset, using table document images, form document images, etc., taking the text content and text position coordinate information of the document image as input, and the structured information of the document image as a label to construct a supervised dataset;

[0043] Step S32, constructing a fine-tuning dataset for the large model of image transaction instruction parsing, using document question and answer data containing location-related information, taking the text content of the document image, the text location coordinate information and the document question and answer questions as input, and the answers to the document question and answer as labels, to construct a supervised fine-tuning dataset;

[0044] Step S33, using the image transaction instruction big model pre-training alignment data set, performing alignment training on the basis of the existing multimodal big model, and obtaining the image transaction instruction basic big model;

[0045] Step S34, using the supervised fine-tuning dataset for image transaction instruction recognition, performing supervised fine-tuning training on the image transaction instruction basic large model to obtain the image transaction instruction parsing large model.

[0046] Furthermore, step S33 also includes:

[0047] Step S2321, establishing a data set of large-scale table images and table structured representation texts, wherein the syntax specification of the table structured representation text is HTML format, and performing text cleaning and filtering;

[0048] Step S2322, establish a large-scale data set of document images and document structured representation texts, and use rules to eliminate low-quality data, wherein the grammatical specification of the document structured representation text is the Markdown format, and the grammatical specification of the table is the HTML format.

[0049] It should be noted that step S3 specifically includes:

[0050] Step S31, traversing the transaction instruction list, using the business-related enterprise knowledge base to search and match the elements of each transaction instruction, and completing the subsidiary elements according to the key elements;

[0051] Step S32, traversing the completed transaction instruction list, using a string comparison algorithm to compare the element value with the entry in the business knowledge base, and correcting the erroneous element caused by the optical character recognition model recognition error;

[0052] Step S33, traverse the completed transaction instruction list, perform disambiguation and repair on repeated conflicting elements, and obtain a final transaction instruction list after correction and completion.

[0053] like Figure 2 , the present application also provides a method for parsing a document image containing investment transaction instructions into an investment transaction instruction list based on a multimodal large language model:

[0054] Step 1000: Use the optical character recognition model to recognize the image:

[0055] Use an optical character recognition model to perform text recognition on images containing investment trading instructions.

[0056] Specifically, step 1000 includes steps 1100-1200, see Figure 3 :

[0057] Step 1100: Image of a document containing transaction instructions.

[0058] An image containing investment trading instructions.

[0059] Step 1200: Optical Character Recognition Model (OCR Model).

[0060] The image containing the investment transaction instruction obtained in step 1100 is input into an optical character recognition model (OCR model).

[0061] Step 1300: Text information and text coordinate information.

[0062] The text information and text position coordinate information generated in step 1200 are sorted and normalized.

[0063] Step 2000: Train a basic large model of image transaction instructions, and train a large model for parsing image transaction instructions.

[0064] Train a large model for parsing image trading instructions. The model will adopt a combination of alignment pre-training and instruction fine-tuning training. First, use formatted text containing tables and documents to align and pre-train the existing multimodal large model to obtain a basic large model of image trading instructions. Then, use the instruction fine-tuning dataset consisting of pictures containing investment trading instructions and corresponding trading instruction annotations on the basic large model of image trading instructions to perform supervised fine-tuning training to obtain a large model for parsing trading instructions.

[0065] For details, see Figure 4 In step 2000, training the basic model of image transaction instructions includes steps 2110-2160.

[0066] Step 2110: Table and document data.

[0067] Get images of tables and documents from the public tables and document datasets, then convert their corresponding formatted text annotations into the specified format of the alignment pre-training dataset, use the alignment pre-training dataset to align and pre-train the existing multimodal large model, and get the basic large model of image transaction instructions. Get images of tables and documents and their corresponding formatted text annotations from the public tables and document datasets. If there are no corresponding annotations, use the document format parsing tool to parse the format and get the corresponding formatted text.

[0068] Step 2120: Convert to a text representation in a specified format.

[0069] The table data and document data obtained in step 2110 are organized and converted into a unified format text structure.

[0070] Step 2130: Instruct fine-tuning the prompt template.

[0071] According to the different types of training samples, corresponding training prompt text templates are written for table images and document images respectively.

[0072] Step 2140: Multimodal big data model aligns the dataset.

[0073] The images, the formatted text corresponding to the images, and the corresponding training prompt text templates are assembled and processed into a multimodal large language model alignment pre-training dataset.

[0074] Step 2150: Large-scale pre-training of multimodal language models.

[0075] Obtain existing large-scale pre-trained multimodal language models.

[0076] Step 2160: Picture transaction instruction basic model.

[0077] The multimodal large language model obtained in step 2150 is used to align the multimodal large language model obtained in step 2140 with the pre-training data set for training to obtain a basic large model of image transaction instructions.

[0078] See also Figure 5 , training the large model for parsing image transaction instructions includes steps 2210-2290.

[0079] Step 2210: Image investment transaction instruction data.

[0080] Organize and filter image transaction instruction data from business data.

[0081] Step 2220: Label element information.

[0082] The transaction instruction elements are annotated in the image transaction instruction data obtained in step 2210, that is, information such as the instruction combination contained in the transaction instruction image and the element type and element value in the combination is indicated.

[0083] Step 2230: Investment transaction instruction list.

[0084] Assemble the instruction combinations and elements to generate the model's target output investment transaction instruction list.

[0085] Step 2240: Serialization tool.

[0086] The investment transaction instruction list obtained in step 2230 is serialized using a JSON list serialization tool.

[0087] Step 2250: Instruct fine-tuning the prompt template.

[0088] Write corresponding prompt text templates based on business scenarios and business target outputs.

[0089] Step 2260: Supervised fine-tuning dataset of multimodal large language model.

[0090] Combine the image, the text and location coordinate information obtained by the optical character recognition model, the serialized investment transaction instruction list and the prompt text template to generate a supervised fine-tuning dataset for the multimodal large language model.

[0091] Step 2270: Picture transaction instruction basic model.

[0092] Obtain the aligned pre-trained image transaction instruction basic large model from step 2160.

[0093] Step 2280: Low rank adaptation layer fine-tuning method.

[0094] The low-rank adaptation layer fine-tuning method is used to perform supervised fine-tuning on the image transaction instruction basic large model obtained in step 2270 and the multimodal large language model supervised fine-tuning dataset obtained in step 2260.

[0095] Step 2290: Parse the large model of image transaction instructions.

[0096] Obtain the trained image transaction instruction parsing model from step 2280.

[0097] Step 3000: Identify transaction instructions from images and texts.

[0098] After training the large model for analyzing picture transaction instructions, the picture containing the transaction instructions to be identified and the text and position information of the picture recognized by the optical character recognition model are input into the large model for analyzing picture transaction instructions to perform transaction instruction recognition.

[0099] For details, see Figure 6 , the step 3000 specifically includes:

[0100] Step 3100: Image of document containing transaction instructions.

[0101] The image containing the trading instructions needs to be adjusted from the original size to a fixed size, and then the image is normalized to obtain the trading instruction image to be identified.

[0102] Step 3200: Text information and text position coordinate information.

[0103] The text and location information obtained from the original transaction instruction image through the optical character recognition model are matched and normalized.

[0104] Step 3300: Task prompt words.

[0105] Select the corresponding task prompt text according to the business scenario, and write in the prompt text information such as the element type to be output by the large model required for the image transaction instruction to parse.

[0106] Step 3400: Parse the large model of image transaction instructions.

[0107] The transaction instruction image, text, location information and task prompt text obtained in steps 3100, 3200 and 3300 are input together into the image transaction instruction parsing model.

[0108] Step 3500: Transaction instruction list.

[0109] The transaction instruction list text is obtained from step 3400, and then the text is deserialized using a corresponding deserialization tool to obtain the transaction instruction list.

[0110] Step 4000: Use the knowledge base for retrieval and matching.

[0111] According to the elements involved in the business, the corresponding business information database is called to search and match each instruction object in the instruction list, and based on the information of some instruction elements, the information of other elements that have not appeared is completed. For example, through the bond code element that has appeared, the database is queried to obtain the bond name corresponding to the bond code, and the bond name is used to fill the missing element value.

[0112] Finally, after completion and improvement, the output content is the final content of this plan and system.

[0113] The present invention also provides a system for analyzing image financial instructions based on a multimodal large model, which is based on the aforementioned method for analyzing image financial instructions based on a multimodal large model. The system includes:

[0114] The multimodal large language model pre-training module uses the image transaction instruction parsing large model position coordinate projection layer pre-training alignment data set to train the existing multimodal large language model to obtain the image transaction instruction basic large model;

[0115] The multimodal large language model has a supervised fine-tuning module, which uses the image transaction instruction parsing large model fine-tuning data set to train the image transaction instruction basic large model to obtain the image transaction instruction parsing large model;

[0116] The instruction recognition module constructs each independent instruction group text into a transaction instruction recognition prompt sentence, inputs the image transaction instruction parsing model, and generates a corresponding transaction instruction list consisting of multiple transaction instructions;

[0117] The knowledge base completion module inputs the transaction instruction list, uses the enterprise business data knowledge base to complete and correct the elements of the instruction, and obtains the final transaction instruction list after correction and completion;

[0118] The continuous feedback learning module collects error samples encountered in the business, analyzes the causes of errors, generates error sample instructions to fine-tune the data set, and iteratively trains the model.

[0119] The large model pre-training module includes a multimodal large language model pre-training module and a multimodal large language model supervised fine-tuning module;

[0120] The multimodal large language model pre-training module analyzes the target usage scenarios of the large model according to the image transaction instructions, constructs table images and document images and their corresponding structured text datasets, and performs module alignment training on the existing multimodal large language model;

[0121] The large language model pre-training module is trained on a multimodal large language model pre-training dataset through mask prediction learning. This pre-training method enables the model to learn the representation of tables and image documents.

[0122] The multimodal large language model supervised fine-tuning module includes an investment instruction processing pipeline and a large language model supervised instruction fine-tuning module;

[0123] The investment instruction processing pipeline cleans and filters the image investment instruction data collected in the business scenario, manually annotates the investment instruction element set list, and assembles and generates an investment instruction fine-tuning data set;

[0124] The multimodal large language model supervised fine-tuning module is trained on the investment instruction fine-tuning dataset through supervised learning. It is necessary to convert the input investment instruction corpus text recognition into an investment instruction element list, and learn to process shared elements and nested elements.

[0125] The instruction recognition module inputs the independent investment instruction corpus into the image transaction instruction recognition language model to obtain a list of investment instruction element sets;

[0126] The knowledge base completion module will go through the list of investment instruction element sets one by one, process each instruction object in the list, and complete the complete elements in the context;

[0127] The continuous feedback learning module collects the error recognition samples encountered by business personnel during the use of the system, re-labels them, generates an investment instruction fine-tuning data set of error samples, uses the large language model instruction fine-tuning module to perform instruction fine-tuning training, and fixes the error recognition problem.

[0128] An embodiment of the present invention further discloses a computer-readable storage medium, which includes a computer program. When the computer program runs on an electronic device, the electronic device executes the above-mentioned method for parsing image financial instructions based on a multimodal large model.

[0129] An embodiment of the present invention further discloses a device comprising a connected processor and a memory, wherein the processor is used to execute a computer program stored in the memory to execute the above-mentioned method for parsing image financial instructions based on a multimodal large model.

[0130] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

[0131] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0132] In the embodiments provided in this application, it should be understood that the disclosed methods can be implemented in other ways without exceeding the spirit and scope of this application. The current embodiment is only an illustrative example and should not be used as a limitation. The specific content given should not limit the purpose of this application. For example, some features can be ignored or not executed.

[0133] The technical means disclosed in the present application are not limited to the technical means disclosed in the above-mentioned implementation methods, but also include technical solutions composed of any combination of the above technical features. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also regarded as the protection scope of this application.

[0134] The above is a detailed introduction to a method for parsing image financial instructions based on a multimodal large model and related devices provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for parsing financial instructions from images based on a multimodal large model, characterized in that: The method comprises: Step S1, inputting an original document image containing an investment transaction instruction, performing text recognition on the original document image using an optical character recognition model, and obtaining a text block list containing text content and text coordinates; Step S2, constructing a picture transaction instruction parsing prompt sentence and a position information vector for the text block list, and inputting the picture transaction instruction parsing prompt sentence and the position information vector into a picture transaction instruction parsing large model, and the picture transaction instruction parsing large model outputs a corresponding transaction instruction list, wherein the transaction instruction list includes all transaction instructions in the picture; Step S3, for the transaction instruction list, each instruction is compared and verified with the transaction business knowledge base, supplemented with additional elements according to business rules, and corrected with text error elements caused by optical character recognition errors according to text features to ensure its completeness and accuracy; Step S4, outputting the complete transaction instruction after the above processing steps.

2. The method for analyzing image financial instructions based on a multimodal large model according to claim 1, characterized in that: The step S2 comprises: Step S21, constructing an investment transaction instruction picture recognition prompt sentence for the text content in the text block list, wherein the investment transaction instruction picture recognition prompt sentence includes one or more of the following: A task description prompt sentence, B target recognition element name list prompt sentence, C element example prompt sentence, D output format sample prompt sentence, and E investment transaction instruction picture text content to be parsed; Among them, the task description prompt statement is used to explain the text source and output format of the recognition task target; the target recognition element name list prompt statement is used to display the element name in a highly readable list; the element example prompt statement contains the possible element values ​​corresponding to the element type; the output format sample prompt statement is used to give an example of the output result format; Step S22, for the text position coordinate information in the text block list, using the position coordinate projection layer to construct a text position coordinate information vector; Step S23, inputting the investment transaction instruction picture recognition prompt sentence and the text position coordinate information vector into the picture transaction instruction parsing model together to generate a transaction instruction list consisting of a corresponding plurality of transaction instructions.

3. The method for analyzing image financial instructions based on a multimodal large model according to claim 2 is characterized in that: The image transaction instruction parsing model is an autoregressive pre-trained language model. The input of the image transaction instruction parsing model is a string text and a text position coordinate information vector. The string text includes a task description, task information, task requirements, and output format, which are used to guide and standardize the generation result of the transaction instruction parsing model; the text position coordinate information vector is used to guide the image transaction instruction parsing model to understand the relative position of the text content on the original image. The steps of constructing the image transaction instruction parsing model specifically include: Step S31, constructing a pre-trained alignment dataset for the position coordinate projection layer of the large model for parsing image transaction instructions, using data containing strong position information association to construct the pre-trained alignment dataset, using table document images, form document images, etc., taking the text content and text position coordinate information of the document image as input, and the structured information of the document image as a label to construct a supervised dataset; Step S32, constructing a fine-tuning dataset for the large model of image transaction instruction parsing, using document question and answer data containing location-related information, taking the text content of the document image, the text location coordinate information and the document question and answer questions as input, and the answers to the document question and answer as labels, to construct a supervised fine-tuning dataset; Step S33, using the image transaction instruction big model pre-training alignment data set, performing alignment training on the basis of the existing multimodal big model, and obtaining the image transaction instruction basic big model; Step S34, using the supervised fine-tuning dataset for image transaction instruction recognition, performing supervised fine-tuning training on the image transaction instruction basic large model to obtain the image transaction instruction parsing large model.

4. The method for analyzing image financial instructions based on a multimodal large model according to claim 3 is characterized in that: Step S33 also includes: Step S2321, establishing a data set of large-scale table images and table structured representation texts, wherein the syntax specification of the table structured representation text is HTML format, and performing text cleaning and filtering; Step S2322, establish a large-scale data set of document images and document structured representation texts, and use rules to eliminate low-quality data, wherein the grammatical specification of the document structured representation text is the Markdown format, and the grammatical specification of the table is the HTML format.

5. The method for analyzing image financial instructions based on a multimodal large model according to claim 1, characterized in that: The step S3 specifically includes: Step S31, traversing the transaction instruction list, using the business-related enterprise knowledge base to search and match the elements of each transaction instruction, and completing the subsidiary elements according to the key elements; Step S32, traversing the completed transaction instruction list, using a string comparison algorithm to compare the element value with the entry in the business knowledge base, and correcting the erroneous element caused by the optical character recognition model recognition error; Step S33, traverse the completed transaction instruction list, perform disambiguation and repair on repeated conflicting elements, and obtain a final transaction instruction list after correction and completion.

6. A picture financial instruction parsing system based on a multimodal large model, characterized in that: The system is applied to the method for parsing picture financial instructions based on a multimodal large model as described in any one of claims 1 to 5, and the system comprises: The multimodal large language model pre-training module uses the image transaction instruction parsing large model position coordinate projection layer pre-training alignment data set to train the existing multimodal large language model to obtain the image transaction instruction basic large model; The multimodal large language model has a supervised fine-tuning module, which uses the image transaction instruction parsing large model fine-tuning data set to train the image transaction instruction basic large model to obtain the image transaction instruction parsing large model; The instruction recognition module constructs each independent instruction group text into a transaction instruction recognition prompt sentence, inputs the image transaction instruction parsing model, and generates a corresponding transaction instruction list consisting of multiple transaction instructions; The knowledge base completion module inputs the transaction instruction list, uses the enterprise business data knowledge base to complete and correct the elements of the instruction, and obtains the final transaction instruction list after correction and completion; The continuous feedback learning module collects error samples encountered in the business, analyzes the causes of errors, generates error sample instructions to fine-tune the data set, and iteratively trains the model.

7. The image financial instruction parsing system based on a multimodal large model according to claim 6 is characterized in that: The large model pre-training module includes a multimodal large language model pre-training module and a multimodal large language model supervised fine-tuning module; The multimodal large language model pre-training module analyzes the target usage scenarios of the large model according to the image transaction instructions, constructs table images and document images and their corresponding structured text datasets, and performs module alignment training on the existing multimodal large language model; The large language model pre-training module is trained on a multimodal large language model pre-training dataset through mask prediction learning. This pre-training method enables the model to learn the representation of tables and image documents. The multimodal large language model supervised fine-tuning module includes an investment instruction processing pipeline and a large language model supervised instruction fine-tuning module; The investment instruction processing pipeline cleans and filters the image investment instruction data collected in the business scenario, manually annotates the investment instruction element set list, and assembles and generates an investment instruction fine-tuning data set; The multimodal large language model supervised fine-tuning module is trained on the investment instruction fine-tuning dataset through supervised learning. It is necessary to convert the input investment instruction corpus text recognition into an investment instruction element list, and learn to process shared elements and nested elements. The instruction recognition module inputs the independent investment instruction corpus into the image transaction instruction parsing model to obtain a list of investment instruction element sets; The knowledge base completion module will go through the list of investment instruction element sets one by one, process each instruction object in the list, and complete the complete elements in the context; The continuous feedback learning module collects the error recognition samples encountered by business personnel during the use of the system, re-labels them, generates an investment instruction fine-tuning data set of error samples, uses the large language model instruction fine-tuning module to perform instruction fine-tuning training, and fixes the error recognition problem.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a computer program, and when the computer program is executed on an electronic device, the electronic device executes the method for parsing image financial instructions based on a multimodal large model as described in any one of claims 1 to 5.

9. A device comprising a processor and a memory connected together, characterized in that: The processor is used to execute the computer program stored in the memory to execute the image financial instruction parsing method based on a multimodal large model as described in any one of claims 1 to 5.

Images