Bill recognition method based on multi-modal large model and related equipment

Abstract

This application discloses a method and related equipment for invoice recognition based on a multimodal large model, used to improve the accuracy of invoice recognition while performing invoice recognition based on a multimodal large model. The method includes: acquiring a mixed invoice image; inputting the mixed invoice image into a pre-trained multimodal large model; the multimodal large model performing the following processing: using a preset invoice type recognition prompt word template to obtain a set of invoice types contained in the mixed invoice image; using a preset structured extraction prompt word template to determine the invoice field content, cross-invoice field attribution, and inter-invoice relationships corresponding to the mixed invoice image, so as to obtain the structured information of each invoice instance corresponding to the mixed invoice image output by the multimodal large model.

Description

Technical Field

[0001] This application relates to the field of bill recognition based on multimodal large models, and more specifically, to a bill recognition method based on multimodal large models, a bill recognition device based on multimodal large models, a bill recognition equipment based on multimodal large models, a computer-readable storage medium, and a computer program product containing instructions. Background Technology

[0002] With the increasing demand for digital processing of invoices, the recognition of mixed invoices (multiple invoices contained in a single image) has become an important application scenario.

[0003] Existing document recognition methods typically employ a multi-stage pipeline approach: first, target detection or layout analysis is used to locate and segment each document instance; then, text direction correction and cropping are performed to obtain a standardized image; and finally, a structured result is generated.

[0004] However, existing technologies are prone to errors in preprocessing steps such as segmentation and correction. Furthermore, the multi-stage pipeline processing method allows detection or recognition errors in earlier stages to be amplified in subsequent stages, resulting in a complex overall process and compromised accuracy of the final structured result. Therefore, existing technologies for document recognition have relatively low accuracy. Summary of the Invention

[0005] This application provides a method, apparatus, device, computer-readable storage medium, and computer program product containing instructions for recognizing invoices based on a multimodal large model, which improves the accuracy of invoice recognition while performing invoice recognition based on a multimodal large model.

[0006] In a first aspect, embodiments of this application provide a bill recognition method based on a multimodal large model, including:

[0007] Acquire mixed-application invoice images, which include invoice images of multiple types;

[0008] The mixed-issue ticket image is input into a pre-trained multimodal large model, which performs the following processing:

[0009] Using a preset ticket type recognition prompt word template, the set of ticket types contained in the mixed ticket image is obtained. The ticket type recognition prompt word template contains multiple predefined ticket types and ticket type semantic information.

[0010] Using a preset structured extraction prompt template, the content of the ticket fields, the cross-ticket field attribution determination, and the inter-ticket relationships corresponding to the mixed-ticket image are determined. Based on the content of the ticket fields, the cross-ticket field attribution determination, and the inter-ticket relationships corresponding to the mixed-ticket image, the structured information of each ticket instance corresponding to the mixed-ticket image output by the multimodal large model is obtained. The structured information of each ticket instance includes the fields and field values ​​of each ticket. The structured extraction prompt template includes the fields and field semantic information corresponding to each ticket type in the ticket type set.

[0011] Secondly, embodiments of this application provide a ticket recognition device based on a multimodal large model, comprising:

[0012] The acquisition unit is used to acquire mixed-application invoice images, wherein the mixed-application invoice images include multiple types of invoice images;

[0013] An input unit is configured to input the mixed-issue ticket image into a pre-trained multimodal large model, wherein the multimodal large model performs the following processing:

[0014] The acquisition unit is also used to obtain the set of bill types contained in the mixed bill image by using a preset bill type recognition prompt word template, wherein the bill type recognition prompt word template contains a variety of predefined bill types and bill type semantic information;

[0015] The acquisition unit is further configured to use a preset structured extraction prompt word template to determine the content of the ticket fields, the cross-ticket field attribution determination, and the inter-ticket relationships corresponding to the mixed-ticket image, and to obtain the structured information of each ticket instance corresponding to the mixed-ticket image output by the multimodal large model based on the content of the ticket fields, the cross-ticket field attribution determination, and the inter-ticket relationships. The structured information of each ticket instance includes the fields and field values ​​of each ticket, and the structured extraction prompt word template includes the fields and field semantic information corresponding to each ticket type in the ticket type set.

[0016] Thirdly, embodiments of this application provide a ticket recognition device based on a multimodal large model, comprising:

[0017] Central processing unit, memory, input / output interfaces, wired or wireless network interfaces, and power supply;

[0018] The memory is either a short-term storage memory or a persistent storage memory;

[0019] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the aforementioned ticket recognition method based on a multimodal large model.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the aforementioned ticket recognition method based on a multimodal large model.

[0021] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to execute the aforementioned ticket recognition method based on a multimodal large model.

[0022] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: This application adopts a two-stage prompt word-driven end-to-end architecture: the first stage uses a preset probing word template to directly obtain the set of bill types from the mixed bill images without physical segmentation and orientation correction; the second stage uses the preset probing word template to determine the bill field content, cross-bill field attribution, and inter-bill relationship of the mixed bill images, so as to load the corresponding field semantic description based on the set of bill types, avoiding the cascading error propagation of traditional multi-stage pipelines. Therefore, the accuracy of bill recognition in this application is high.

[0023] Accordingly, the multimodal large-scale bill recognition device, multimodal large-scale bill recognition equipment, computer-readable storage medium, and computer program product containing instructions provided in this application also have the above-mentioned technical effects. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the architecture of a ticket recognition system based on a multimodal large model disclosed in an embodiment of this application;

[0025] Figure 2 This is a flowchart illustrating a bill recognition method based on a multimodal large model disclosed in an embodiment of this application;

[0026] Figure 3 This is a flowchart illustrating another invoice recognition method based on a multimodal large model disclosed in an embodiment of this application;

[0027] Figure 4 This is a flowchart illustrating a multimodal large model fine-tuning training method disclosed in an embodiment of this application;

[0028] Figure 5 This is a schematic diagram of the structure of a ticket recognition device based on a multimodal large model disclosed in an embodiment of this application;

[0029] Figure 6 This is a schematic diagram of the structure of a ticket recognition device based on a multimodal large model disclosed in an embodiment of this application. Detailed Implementation

[0030] This application provides a method, apparatus, device, computer-readable storage medium, and computer program product containing instructions for recognizing invoices based on a multimodal large model, which improves the accuracy of invoice recognition while performing invoice recognition based on a multimodal large model.

[0031] Please see Figure 1 The architecture of the invoice recognition system based on a multimodal large model in this application embodiment includes:

[0032] A multimodal large-scale model-based invoice recognition device 101 and a client 102 are used. When performing invoice recognition based on the multimodal large-scale model, the invoice recognition device 101 can connect to the client 102. The multimodal large-scale model-based invoice recognition device 101 can obtain mixed invoice images sent by the client 102, and input the mixed invoice images into a pre-trained multimodal large-scale model. The multimodal large-scale model performs the following processing: using a preset invoice type recognition prompt word template, it obtains the set of invoice types contained in the mixed invoice images; then, using a preset structured extraction prompt word template, it determines the invoice field content, cross-invoice field attribution determination, and inter-invoice relationships corresponding to the mixed invoice images; and based on the invoice field content, cross-invoice field attribution determination, and inter-invoice relationships corresponding to the mixed invoice images, it obtains the structured information of each invoice instance corresponding to the mixed invoice images output by the multimodal large-scale model. Finally, it returns the structured information of each invoice instance corresponding to the mixed invoice images to the client 102.

[0033] based on Figure 1 Please refer to the document recognition system based on a multimodal large model shown below. Figure 2 , Figure 2 This is a flowchart illustrating a bill recognition method based on a multimodal large model disclosed in an embodiment of this application. The method includes:

[0034] 201. Obtain mixed-issue invoice images, which include invoice images of multiple types.

[0035] In one alternative implementation, mixed-image document refers to a single image containing multiple documents of different types. Specifically, it refers to differences in the format, size, or placement of the document images. The placement method includes the placement direction and spatial layout. The placement direction refers to whether the documents are placed vertically or horizontally. The spatial layout refers to the physical overlap, partial obscuring, juxtaposition, and / or staggering of the documents. The format refers to the type of document image, such as invoices or tickets.

[0036] It is important to understand that, due to the variable number of tickets in mixed ticket images, as well as differences in layout, size, and placement, the situation is quite complex, making it difficult to meet the traditional single-ticket recognition's prerequisite of "clear boundaries and regular layout."

[0037] 202. Input the mixed-issue image into the pre-trained multimodal large model. The multimodal large model performs the following processing: using the preset document type recognition prompt word template, obtain the set of document types contained in the mixed-issue image. The document type recognition prompt word template contains multiple predefined document types and document type semantic information.

[0038] In one optional implementation, a preset document type recognition prompt word template can be used to perform overall semantic analysis on the mixed document image to obtain the set of document types contained in the mixed document image. The document type recognition prompt word template contains multiple predefined document types and their semantic information. The preset document type recognition prompt word template refers to a system-pre-built instruction template containing multiple document types and their format semantic features (e.g., VAT invoices are vertical A4 format, train tickets are horizontal strips), used to guide the multimodal large model in recognizing the types of documents contained in the mixed image without requiring physical image segmentation. The document type set refers to a list of all document types contained in the mixed document image (e.g., [VAT general invoice, train ticket]), serving as the input for the second-stage field extraction.

[0039] 203. Using a preset structured extraction prompt template, determine the field content of the mixed-item images, the cross-item field attribution determination, and the relationship between the items. Based on the field content of the mixed-item images, the cross-item field attribution determination, and the relationship between the items, obtain the structured information of each item instance corresponding to the mixed-item images output by the multimodal large model. The structured information of each item instance includes the fields and field values ​​of each item. The structured extraction prompt template includes the fields and field semantic information corresponding to each item type in the item type set.

[0040] In one optional implementation, a preset structured extraction prompt template can be used to perform overall semantic analysis and spatial understanding on the mixed ticket image. This determines the ticket field content corresponding to the mixed ticket image, cross-ticket field attribution, and inter-ticket relationships. Based on the ticket field content, cross-ticket field attribution, and inter-ticket relationships, the structured information of each ticket instance corresponding to the mixed ticket image output by the multimodal large model is obtained. The preset structured extraction prompt template refers to an instruction template dynamically generated based on the ticket type set identified in the first stage, containing field definitions for each type (e.g., invoices contain "amount," tickets contain "ticket price"), used to guide the model in extracting specific field values. Overall semantic analysis and spatial understanding of the mixed ticket image means that the model uses the entire mixed image as a unified input, simultaneously understanding the semantic meaning (OCR) of the text content and the geometric positional relationship of each text block / ticket in the image (e.g., "¥280" located in a horizontal strip area belongs to a train ticket), without the need for pre-segmentation or rotation correction. Ticket field content refers to the specific information recorded on each ticket (e.g., amount, date, train number). Cross-document field attribution determination refers to determining which specific document a extracted field value belongs to (e.g., attributing "¥280" to a train ticket rather than a neighboring invoice). Document inter-relationships refer to the spatial relationships between multiple documents (e.g., overlapping, obscuring, juxtaposition). The structured information of each document instance refers to the final output JSON format data, where each document is treated as an independent object, containing its document type and all key-value pairs (e.g., {Type: VAT invoice, Amount: ¥5000}).

[0041] Please refer to the details. Figure 3 , Figure 3 This is a flowchart illustrating another invoice recognition method based on a multimodal large model disclosed in an embodiment of this application. Figure 3It can be seen that a two-stage prompt word-driven structured extraction of mixed-item tickets can be achieved based on a multimodal large model: The first stage assembles the mixed-item ticket image with pre-set semantic descriptions of each ticket type (i.e., ticket type 1: semantic description of ticket type 1; ticket type 2: semantic description of ticket type 2... ticket type n: semantic description of ticket type n) into a ticket type recognition prompt word template. Using this template, a ticket type set is obtained. The second stage automatically retrieves the semantic descriptions of each field for the corresponding ticket type based on this set (i.e., ticket type 1: field 1: field... field 2 ... 1. Semantic description; Field 2: Semantic description of field 2...... Field m1: Semantic description of field m1;”; Ticket type 2: “Field 1: Semantic description of field 1; Field 2: Semantic description of field 2...... Field m2: Semantic description of field m2;” and assembled into a structured extraction prompt word template. Using the preset structured extraction prompt word template, the ticket field content corresponding to the mixed ticket image, the cross-ticket field attribution judgment, and the association relationship between tickets are determined. Finally, the structured information of each ticket instance corresponding to the mixed ticket image is obtained. No explicit segmentation or orientation correction of the image is required throughout the process.

[0042] It is worth mentioning that this application adopts a two-stage prompt word-driven end-to-end architecture: the first stage uses a preset prompt word template to directly obtain the set of ticket types from the mixed ticket images, without the need for physical segmentation and orientation correction; the second stage uses the preset prompt word template to determine the ticket field content, cross-ticket field attribution, and inter-ticket relationships corresponding to the mixed ticket images, so as to load the corresponding field semantic description based on the ticket type set, avoiding the cascading error propagation of traditional multi-stage pipelines. Therefore, the ticket recognition accuracy of this application is high. Secondly, to address the problem of low fault tolerance in blurry and occluded scenes of existing pixel-level extraction methods, this application adopts the cross-modal semantic understanding capability of multimodal large models. Through overall layout analysis combined with contextual semantic reasoning, it corrects cross-ticket attribution errors and improves the recognition accuracy in complex mixed scenes. Furthermore, to address the limitation of traditional technologies that can only extract text coordinates and cannot understand logical relationships, this application uses a large model to determine the ticket field content, cross-ticket field attribution, and inter-ticket relationships, which can understand the inherent relationships of text, images, and other elements, and achieve end-to-end deep structured extraction.

[0043] In one optional implementation, before inputting the mixed-item ticket images into a pre-trained multimodal large model, the method further includes: supervising the original multimodal large model using target mixed-item ticket image samples to obtain a multimodal large model trained in the first stage, wherein the target mixed-item ticket image samples are labeled with structured information of each ticket instance sample; generating multiple candidate outputs corresponding to the target mixed-item ticket image samples based on the multimodal large model trained in the first stage, calculating the field-level continuous reward value of each candidate output, and using a target multidimensional advantage strategy optimization algorithm to optimize the model strategy of the multimodal large model trained in the first stage based on the continuous reward difference to obtain the... The multimodal large model is completed in two stages of training. Field-level continuous reward values ​​are determined based on field accuracy, which is the ratio of the number of correct fields to the total number of fields. After generating multiple candidate outputs corresponding to mixed-item image samples in the target mixed-item image sample subset based on the multimodal large model trained in the second stage, the discrete reward value of each candidate output is calculated. A target multidimensional advantage strategy optimization algorithm is then used to optimize the model strategy based on the discrete reward differences, resulting in a pre-trained multimodal large model. The discrete reward value is determined based on complete accuracy, and a positive reward is given when all fields and field values ​​of each item instance are completely matched.

[0044] It is important to understand that the target mixed-format document image sample includes multiple types of document image samples. A target mixed-format document image sample refers to a single image sample containing multiple document images of different types. Specifically, this means that the document image samples differ in layout, size, or placement. Placement includes placement direction and spatial layout. Placement direction refers to whether the documents are placed vertically or horizontally. Spatial layout refers to the physical overlap, partial obstruction, juxtaposition, and / or staggering of the documents. Layout refers to the type of document image sample, such as invoices, tickets, etc.

[0045] Specifically, the following model training strategy can be used to train the multimodal large model in multiple stages. The first stage employs SFT (Supervised Fine-Tuning) alignment tasks. By constructing instruction samples that provide an overall understanding and structured output of mixed-issue documents, the model learns to identify document types and extract fields without explicitly segmenting the documents. The second stage uses MDAPO (Target Multidimensional Advantage Optimization) to enhance task capabilities and improve accuracy. Addressing the issues of unclear field attribution and severe cross-document field interference in mixed-issue documents, a continuous field-level reward (field accuracy) is introduced to finely constrain the model output, preventing reward collapse and stabilizing the optimization process. The third stage continues MDAPO reinforcement learning on specific business data (mixed-issue image samples within the target mixed-issue image sample subset). A discrete reward (complete accuracy) for perfectly matching the overall structure of the mixed-issue documents is further introduced, strengthening the model's learning of multi-document structural consistency, field completeness, and cross-document differentiation capabilities. In this stage, a discrete reward (complete accuracy) is used as the reward function to further improve accuracy.

[0046] Thus, the first stage performs SFT supervised training to establish the basic capabilities of bill type recognition and field extraction; the second stage performs the first MDAPO reinforcement learning and introduces field-level continuous rewards (based on field accuracy) to improve field attribution accuracy and avoid reward collapse; the third stage performs the second MDAPO reinforcement learning and introduces discrete rewards (based on complete accuracy) to enhance the consistency of multi-bill structure and field integrity, and finally obtains a pre-trained multimodal large model.

[0047] In an optional implementation, before generating multiple candidate outputs corresponding to the target mixed-item image samples based on the multimodal large model trained in the first stage, the method further includes: generating multiple candidate outputs corresponding to each mixed-item image sample based on the multimodal large model trained in the first stage, calculating the target reward value of each candidate output, and determining mixed-item image samples whose error rates for the content of the item field, the determination of cross-item field attribution, and the correlation between items are greater than a preset error rate threshold as high-value samples. Among the high-value samples, the top N high-value samples with the largest difference in target reward among the candidate outputs are preferentially selected as the target mixed-item image samples.

[0048] Specifically, this application improves the MDAPO algorithm training as follows: First, compared to the original DAPO method which only filters reward difference samples, this application prioritizes samples with high field attribution error rates, severe inter-ticket confusion, or significant structural errors as high-value samples in mixed-ticket samples. Furthermore, it prioritizes the top N samples with the largest reward differences for advantage learning, thereby improving training efficiency. Second, considering the characteristics of mixed-ticket structuring tasks involving multiple fields, multiple tickets, and multi-step inference chains, it replaces token-level policy gradient loss with sequence-level policy gradient loss, using the overall structured output sequence as the optimization unit. This strengthens the model's constraints on overall structural correctness and cross-field consistency, thereby improving training convergence speed and test accuracy in complex mixed-ticket scenarios.

[0049] For the MDAPO objective function (the mathematical optimization objective function of the objective multidimensional advantage strategy optimization algorithm), please refer to Formula 1:

[0050]

[0051] Formula 1

[0052] Thus, based on the improved DAPO method, by dynamically sampling high-value samples with a field attribution error rate greater than a preset threshold, and further selecting the top N samples with the largest differences in candidate output rewards for advantage learning, the model training efficiency and structure extraction accuracy can be further improved.

[0053] In one optional implementation, the candidate outputs exist in the form of a list. Calculating the field-level continuous reward value for each candidate output includes: uniformly converting the basic data types in the standard answer list and each candidate output list to the target data type; aligning the order of the ticket instances in the standard answer list and each candidate output list using a similarity algorithm; verifying whether each field value of each ticket instance in each candidate output list belongs to the correct ticket instance type; comparing the field differences between the aligned standard answer list and each candidate output list; and using the ratio of the number of correctly assigned fields in each candidate output list to the total number of fields as the field-level continuous reward value for each candidate output list.

[0054] Specifically, the reward function design in this application can be as follows: recursively convert the basic data types of the standard answer list and the candidate output list into string data types to avoid misjudgments caused by differences between numerical and string types; align the order of ticket instances in the list using a similarity algorithm to avoid errors in field accuracy calculation due to inconsistent order; verify whether each field value belongs to the correct ticket instance type for mixed ticket scenarios, and penalize incorrect assignments; compare the differences between aligned fields, and use the ratio of the number of correctly assigned fields to the total number of fields as the field-level continuous reward value.

[0055] For the reward function formula (the formula for calculating the continuous reward value at the field level), please refer to Formula 2:

[0056]

[0057] Formula 2

[0058] In this way, the field accuracy reward function design can alleviate the reward collapse problem in reinforcement learning and improve training stability.

[0059] In one optional implementation, the order of ticket instances in the standard answer list and each candidate output list is aligned using a similarity algorithm, including: aligning the standard answer list and each candidate output list based on a dictionary similarity algorithm so that the same ticket instance is at the same index position in both the standard answer list and each candidate output list.

[0060] Specifically, the similarity of field overlap between each ticket instance in the standard answer list and the candidate output list can be calculated to construct a similarity matrix. Then, the similarity matrix is ​​converted into a cost matrix, and the Hungarian algorithm is used to solve for minimum-cost matching, obtaining the optimal correspondence between the ticket instances in the standard answer list and the candidate output list. Finally, the candidate output list is reorganized based on the optimal correspondence, ensuring that the same ticket instance is at the same index position in both the standard answer list and the candidate output list.

[0061] In this way, aligning the standard answer list and the candidate output list based on a dictionary similarity algorithm ensures that instances of the same invoice are at the same index position. This avoids misalignment of field-level comparisons caused by inconsistencies between the model's output order and the labeling order (such as mismatched invoice and ticket fields), guaranteeing the accuracy of subsequent field accuracy calculations. This alignment mechanism provides a reliable benchmark for the reward function, enabling the model to obtain accurate gradient feedback, thereby improving training efficiency and recognition performance.

[0062] In one alternative implementation, before supervising the training of the original multimodal large model using the target mixed-issue ticket image samples, the method further includes: acquiring the original mixed-issue ticket image samples; spatially combining and rearranging the ticket instances of the original mixed-issue ticket image samples through ticket-level sampling; and recombining the field set of the original mixed-issue ticket image samples across tickets through field-level sampling to construct expanded samples; and obtaining the target mixed-issue ticket image samples based on the original mixed-issue ticket image samples and the expanded samples.

[0063] It is important to understand that original mixed-issue image samples include multiple types of original issue image samples. Original mixed-issue image samples refer to a single image sample containing multiple original issue images of different types. Specifically, this means that the original issue image samples differ in layout, size, or placement. Placement includes placement direction and spatial layout. Placement direction refers to whether the issue images are placed vertically or horizontally. Spatial layout refers to the physical overlap, partial obstruction, juxtaposition, and / or staggering of the issue images. Layout refers to the type of original issue image sample, such as invoices, tickets, etc.

[0064] Specifically, document-level sampling refers to spatially combining and rearranging document instances in the original mixed document image sample, generating multiple document spatial layout variants through random arrangement, overlap control, rotation transformation, and scaling transformation. Field-level sampling refers to cross-document recombination of the field set in the original mixed document image sample, extracting the field regions and inserting them into blank or overlapping areas of other documents to simulate cross-document interference scenarios.

[0065] Optionally, this application may also employ a model-assisted iterative annotation process to reduce the cost of manual annotation. Specifically, model annotation trained on small batches of data assists manual annotation, and iterative difference comparison improves quality inspection efficiency.

[0066] In this way, this application can balance the distribution of samples across multiple categories and retain difficult examples containing blurred, occluded, and missing field features to construct target mixed-format ticket image samples. Secondly, by using two-level sampling to augment data under limited sample conditions, it can reduce annotation costs and improve the model's generalization ability to complex mixed-format scenarios while ensuring sample diversity and category balance. Finally, this application possesses strong generalization and few-sample learning capabilities, allowing adaptation to new ticket styles with only a small number of samples. Its strong adaptability significantly reduces scenario adaptation costs.

[0067] To facilitate understanding of the embodiments of this application, the multimodal large model fine-tuning training method of this application is described below. Please refer to [link / reference] for details. Figure 4 , Figure 4 This is a flowchart illustrating a multimodal large model fine-tuning training method disclosed in an embodiment of this application. Figure 4This paper presents a closed-loop training technique for mixed-document recognition models, comprising four stages: data preparation, training set construction, model training, and model evaluation. Specifically, in the data preparation stage, small-batch manual annotation is performed on mixed-document image samples to train the SFT model. This model is then used to assist in labeling. Finally, an iterative process of manual quality inspection through difference comparison is implemented to reduce annotation costs, resulting in original mixed-document image samples. In the training set construction stage, data quality enhancement is first used to diversify image quality and balance various document types. Then, document-level sampling spatially combines and rearranges document instances, and field-level sampling performs cross-document recombination of field sets to construct expanded samples. The multimodal large-scale model training phase comprises three stages. The first stage uses mixed-image samples of invoices labeled with structured information for supervised training of the original multimodal large-scale model, establishing its basic recognition capabilities. The second stage calculates continuous reward values ​​based on field accuracy and employs a multidimensional advantage strategy optimization algorithm to perform reinforcement learning optimization on the first-stage model, improving field-level recognition accuracy. The third stage calculates discrete reward values ​​based on complete accuracy and uses the multidimensional advantage strategy optimization algorithm to perform reinforcement learning optimization on the second-stage model, ensuring the complete correctness of the structured information on the entire invoice. In the multimodal large-scale model evaluation phase, the multi-index evaluation results are fed back to the data preparation phase, forming an iterative optimization closed loop between data preparation and multimodal large-scale model training.

[0068] Thus, this application, through an end-to-end two-stage prompt-driven architecture, unifies the identification of invoice types and the structured extraction of fields into a single or a few inference processes within a multimodal large model, eliminating the need for explicit invoice instance segmentation, orientation correction, and format judgment, which are prone to introducing errors. Secondly, it achieves cross-field attribution determination through overall semantic analysis and spatial understanding, replacing manual rule templates with predefined invoice type semantic information and field semantic descriptions. Furthermore, it employs SFT and MDAPO multi-stage training strategies to optimize the model's generalization ability. This solves the problems of progressively amplified errors in multi-stage pipelines, insufficient stability in overlapping and occluded scenarios, high adaptation costs for new invoice types, and low efficiency of multi-model concatenation in existing technologies. This achieves the technical goals of improving the accuracy of identification in complex mixed-use scenarios, reducing scenario adaptation costs, and improving the long-term maintenance efficiency of the system.

[0069] For further details, please refer to Figure 5 One embodiment of the multimodal large model-based ticket recognition device in this application includes:

[0070] The acquisition unit is used to acquire mixed-application invoice images, wherein the mixed-application invoice images include multiple types of invoice images;

[0071] An input unit is configured to input the mixed-issue ticket image into a pre-trained multimodal large model, wherein the multimodal large model performs the following processing:

[0072] The acquisition unit is also used to obtain the set of bill types contained in the mixed bill image by using a preset bill type recognition prompt word template, wherein the bill type recognition prompt word template contains a variety of predefined bill types and bill type semantic information;

[0073] The acquisition unit is further configured to use a preset structured extraction prompt word template to determine the content of the ticket fields, the cross-ticket field attribution determination, and the inter-ticket relationships corresponding to the mixed-ticket image, and to obtain the structured information of each ticket instance corresponding to the mixed-ticket image output by the multimodal large model based on the content of the ticket fields, the cross-ticket field attribution determination, and the inter-ticket relationships. The structured information of each ticket instance includes the fields and field values ​​of each ticket, and the structured extraction prompt word template includes the fields and field semantic information corresponding to each ticket type in the ticket type set.

[0074] In an optional implementation, the ticket recognition device based on a multimodal large model further includes a training unit, which can be used for:

[0075] The original multimodal large model is trained under supervision using target mixed-label ticket image samples to obtain the first-stage trained multimodal large model. The target mixed-label ticket image samples are labeled with the structured information of each ticket instance sample.

[0076] After generating multiple candidate outputs corresponding to the target mixed-issue ticket image samples based on the multimodal large model trained in the first stage, the field-level continuous reward value of each candidate output is calculated. Then, the target multidimensional advantage strategy optimization algorithm is used to optimize the model strategy of the multimodal large model trained in the first stage based on the difference in continuous rewards, so as to obtain the multimodal large model trained in the second stage. The field-level continuous reward value is determined based on the field accuracy, which is the ratio of the number of correct fields to the total number of fields.

[0077] After generating multiple candidate outputs corresponding to the mixed-item image samples in the target mixed-item image sample subset based on the multimodal large model trained in the second stage, the discrete reward value of each candidate output is calculated. Then, the target multidimensional advantage strategy optimization algorithm is used to optimize the model strategy of the multimodal large model trained in the second stage according to the discrete reward difference, so as to obtain the pre-trained multimodal large model. The discrete reward value is determined based on the complete accuracy. A positive reward is given when all fields and field values ​​of each ticket instance are completely matched.

[0078] In one alternative implementation, the acquisition unit may also be used for:

[0079] Based on the multimodal large model trained in the first stage, multiple candidate outputs are generated for each mixed-issue ticket image sample, and the target reward value of each candidate output is calculated.

[0080] Mixed document image samples with error rates greater than a preset error rate threshold for candidate output document field content, cross-document field attribution determination, and inter-document association relationship are identified as high-value samples.

[0081] Among the high-value samples, the top N high-value samples with the largest differences in target reward among the candidate outputs are preferentially selected as the target mixed-issue ticket image samples.

[0082] In one alternative implementation, the training unit can also be used for:

[0083] The basic data types of the standard answer list and each candidate output list are uniformly converted to the target data type;

[0084] The order of the ticket instances in the standard answer list and the candidate output lists is aligned using a similarity algorithm.

[0085] Verify that the values ​​of each field of each ticket instance in each candidate output list belong to the correct ticket instance type;

[0086] The field differences between the aligned standard answer list and each candidate output list are compared, and the ratio of the number of correctly assigned fields in each candidate output list to the total number of fields is used as the field-level continuous reward value of each candidate output list.

[0087] In one alternative implementation, the training unit can also be used for:

[0088] Align the standard answer list with the candidate output lists based on a dictionary similarity algorithm, so that the same ticket instance is at the same index position in both the standard answer list and the candidate output lists.

[0089] In one alternative implementation, the acquisition unit may also be used for:

[0090] Obtain original mixed-issue invoice image samples;

[0091] The original mixed-issue invoice image sample is spatially combined and rearranged by invoice-level sampling, and the field set of the original mixed-issue invoice image sample is recombined across invoices by field-level sampling to construct an expanded sample.

[0092] The target mixed-issue image sample is obtained based on the original mixed-issue image sample and the expanded sample.

[0093] For further details, please refer to Figure 6 One embodiment of the multimodal large model-based ticket recognition device in this application includes:

[0094] Central processing unit 601, memory 605, input / output interface 604, wired or wireless network interface 603, and power supply 602.

[0095] Memory 605 is either a short-term storage memory or a persistent storage memory;

[0096] The central processing unit 601 is configured to communicate with the memory 605 and execute instructions stored in the memory 605 to perform the aforementioned operations. Figure 2 The method in the illustrated embodiment.

[0097] Furthermore, embodiments of this application also provide a computer-readable storage medium, which includes instructions that, when executed on a computer, cause the computer to perform the aforementioned... Figure 2 The method in the illustrated embodiment.

[0098] Furthermore, embodiments of this application also provide a computer program product containing instructions, which, when run on a computer, causes the computer to perform the aforementioned... Figure 2 The method in the illustrated embodiment.

[0099] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0100] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0101] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0102] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0103] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0104] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A bill identification method based on a multi-modal large model, characterized in that, include: Acquire mixed-application invoice images, which include invoice images of multiple types; The mixed-issue ticket image is input into a pre-trained multimodal large model, which performs the following processing: Using a preset ticket type recognition prompt word template, the set of ticket types contained in the mixed ticket image is obtained. The ticket type recognition prompt word template contains multiple predefined ticket types and ticket type semantic information. Using a preset structured extraction prompt template, the content of the ticket fields, the cross-ticket field attribution determination, and the inter-ticket relationships corresponding to the mixed-ticket image are determined. Based on the content of the ticket fields, the cross-ticket field attribution determination, and the inter-ticket relationships corresponding to the mixed-ticket image, the structured information of each ticket instance corresponding to the mixed-ticket image output by the multimodal large model is obtained. The structured information of each ticket instance includes the fields and field values ​​of each ticket. The structured extraction prompt template includes the fields and field semantic information corresponding to each ticket type in the ticket type set.

2. The method of claim 1, wherein, Before inputting the mixed-issue ticket image into a pre-trained multimodal large model, the method further includes: The original multimodal large model is trained under supervision using target mixed-label ticket image samples to obtain the first-stage trained multimodal large model. The target mixed-label ticket image samples are labeled with the structured information of each ticket instance sample. After generating multiple candidate outputs corresponding to the target mixed-issue ticket image samples based on the multimodal large model trained in the first stage, the field-level continuous reward value of each candidate output is calculated. Then, the target multidimensional advantage strategy optimization algorithm is used to optimize the model strategy of the multimodal large model trained in the first stage based on the difference in continuous rewards, so as to obtain the multimodal large model trained in the second stage. The field-level continuous reward value is determined based on the field accuracy, which is the ratio of the number of correct fields to the total number of fields. After generating multiple candidate outputs corresponding to the mixed-item image samples in the target mixed-item image sample subset based on the multimodal large model trained in the second stage, the discrete reward value of each candidate output is calculated. Then, the target multidimensional advantage strategy optimization algorithm is used to optimize the model strategy of the multimodal large model trained in the second stage according to the discrete reward difference, so as to obtain the pre-trained multimodal large model. The discrete reward value is determined based on the complete accuracy. A positive reward is given when all fields and field values ​​of each ticket instance are completely matched.

3. The method according to claim 2, characterized in that, Before generating multiple candidate outputs corresponding to the target mixed-signature ticket image samples based on the multimodal large model trained in the first stage, the method further includes: Based on the multimodal large model trained in the first stage, multiple candidate outputs are generated for each mixed-issue ticket image sample, and the target reward value of each candidate output is calculated. Mixed document image samples with error rates greater than a preset error rate threshold for candidate output document field content, cross-document field attribution determination, and inter-document association relationship are identified as high-value samples. Among the high-value samples, the top N high-value samples with the largest differences in target reward among the candidate outputs are preferentially selected as the target mixed-issue ticket image samples.

4. The method according to claim 2, characterized in that, The candidate outputs exist in list form, and the calculation of the field-level continuous reward value for each candidate output includes: The basic data types of the standard answer list and each candidate output list are uniformly converted to the target data type; The order of the ticket instances in the standard answer list and the candidate output lists is aligned using a similarity algorithm. Verify that the values ​​of each field of each ticket instance in each candidate output list belong to the correct ticket instance type; The field differences between the aligned standard answer list and each candidate output list are compared, and the ratio of the number of correctly assigned fields in each candidate output list to the total number of fields is used as the field-level continuous reward value of each candidate output list.

5. The method according to claim 4, characterized in that, The step of aligning the order of the standard answer list with the order of the ticket instances in each candidate output list using a similarity algorithm includes: Align the standard answer list with the candidate output lists based on a dictionary similarity algorithm, so that the same ticket instance is at the same index position in both the standard answer list and the candidate output lists.

6. The method according to claim 2, characterized in that, Before performing supervised training of the original multimodal large model using target mixed-signature image samples, the method further includes: Obtain original mixed-issue invoice image samples; The original mixed-issue invoice image sample is spatially combined and rearranged by invoice-level sampling, and the field set of the original mixed-issue invoice image sample is recombined across invoices by field-level sampling to construct an expanded sample. The target mixed-issue image sample is obtained based on the original mixed-issue image sample and the expanded sample.

7. A ticket recognition device based on a multimodal large model, characterized in that, include: The acquisition unit is used to acquire mixed-application invoice images, wherein the mixed-application invoice images include multiple types of invoice images; An input unit is configured to input the mixed-issue ticket image into a pre-trained multimodal large model, wherein the multimodal large model performs the following processing: The acquisition unit is also used to obtain the set of bill types contained in the mixed bill image by using a preset bill type recognition prompt word template, wherein the bill type recognition prompt word template contains a variety of predefined bill types and bill type semantic information; The acquisition unit is further configured to use a preset structured extraction prompt word template to determine the content of the ticket fields, the cross-ticket field attribution determination, and the inter-ticket relationships corresponding to the mixed-ticket image, and to obtain the structured information of each ticket instance corresponding to the mixed-ticket image output by the multimodal large model based on the content of the ticket fields, the cross-ticket field attribution determination, and the inter-ticket relationships. The structured information of each ticket instance includes the fields and field values ​​of each ticket, and the structured extraction prompt word template includes the fields and field semantic information corresponding to each ticket type in the ticket type set.

8. A ticket recognition device based on a multimodal large model, characterized in that, include: Central processing unit and memory; The memory is either a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 6.

10. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 6.

Images