Business document processing method and device based on multi-modal model, equipment, storage medium and product

Abstract

The application discloses a business document processing method and device based on a multi-modal model, equipment, a storage medium and a product, relates to the technical field of business document auditing, and comprises the following steps: acquiring a business document image; inputting the business document image into a pre-trained and optimized VLM model, performing semantic extraction and coarse positioning through the VLM model, obtaining a semantic extraction result and a corresponding prediction bounding box as a coarse positioning result, and training and optimizing the VLM model by using an end-to-end GRPO reinforcement learning algorithm; and calling an OCR model to finely correct the coarse positioning result output by the VLM model, and obtaining a final positioning result. The application realizes the organic fusion of the VLM semantic understanding capability and the OCR accurate positioning capability, effectively solves the problem that the VLM model is not accurate in positioning in a complex layout and multi-value repeated field scene, and significantly improves the accuracy of field positioning and the robustness of recognition results.

Description

Technical Field

[0001] This application relates to the field of business document review technology, and in particular to a business document processing method, apparatus, equipment, storage medium and product based on a multimodal model. Background Technology

[0002] In the field of cross-border business document review, existing technologies mainly rely on a combination of OCR (Optical Character Recognition) technology and large language models for document information extraction. OCR technology is responsible for recognizing the text content in document images and converting it into editable text, which is then used by the large language model for semantic understanding and field extraction based on the recognition results. In recent years, with the development of VLM (Vision-Language Model) technology, some research has begun to explore its application to document information extraction tasks. Leveraging its combined visual and language understanding capabilities, VLM directly identifies and outputs key field information from documents based on natural language prompts, aiming to improve the understanding of complex layouts and unstructured documents.

[0003] However, the aforementioned existing technologies still have significant shortcomings: On the one hand, traditional OCR technology lacks semantic understanding capabilities, making it unable to accurately identify the business meaning of professional terms and fields in documents. It also has poor adaptability to complex layouts, easily causing recognition errors and row-column correspondence confusion in scenarios such as complex layouts and multi-column tables, affecting the accuracy of information extraction. On the other hand, OCR positioning lacks semantic guidance and cannot perform intelligent filtering based on semantic needs, resulting in positioning results that do not match actual business needs.

[0004] On the other hand, while existing VLM models possess some semantic understanding capabilities, their training objectives are primarily focused on optimizing semantic extraction tasks, with insufficient training in location localization capabilities. In joint tasks requiring simultaneous output of extraction results and location coordinates, semantic understanding and location awareness are not coordinated, making it difficult to meet business requirements in terms of localization accuracy. This is especially true in document scenarios with complex structures and repetitive fields, where the model is prone to locating the wrong target area. Furthermore, the traditional IoU (Intersection over Union) reward function used in existing models during training often results in a reward value of 0 when the model's initial localization ability is weak, failing to provide an effective training signal and hindering the improvement of the model's localization capabilities. Summary of the Invention

[0005] The main purpose of this application is to provide a business document processing method, device, equipment, storage medium and product based on a multimodal model, which aims to achieve joint optimization of semantic understanding and positioning capabilities, and improve the accuracy and robustness of document information extraction.

[0006] To achieve the above objectives, this application proposes a business document processing method based on a multimodal model, the method comprising: Obtain images of business documents; The business document image is input into a pre-trained and optimized VLM model. The VLM model is used to perform semantic extraction and coarse localization to obtain the semantic extraction result and the corresponding predicted bounding box as the coarse localization result. The VLM model is trained and optimized using the end-to-end GRPO reinforcement learning algorithm. The coarse localization result output by the VLM model is finely corrected by calling the OCR model to obtain the final localization result.

[0007] In one embodiment, the step of calling the OCR model to refine the coarse localization result output by the VLM model to obtain the final localization result includes: The OCR model is invoked to perform full text detection and recognition on the business document image, and multiple text boxes and their corresponding text content and precise coordinates are obtained. Using the field values ​​in the semantic extraction results as the matching key, a set of candidate text boxes with consistent text content is filtered out from the text boxes. Calculate the geometric distance between each text box in the candidate text box set and the predicted bounding box; The text box with the smallest geometric distance is selected, and its precise coordinates are used to replace the predicted bounding box as the final positioning result.

[0008] In one embodiment, the VLM model training optimization step includes: Obtain a training dataset and a training base model. The training dataset is constructed based on original document image data containing labeled field values ​​and their true bounding box coordinates. Based on the training dataset, prompt words containing a fixed extraction instruction template and field variables to be extracted are input into the training base model. The training base model is constrained to output the semantic extraction results of the field and the corresponding predicted bounding box coordinates in a predetermined JSON format to obtain the initial prediction results. The VLM model is trained with all parameters using the end-to-end GRPO reinforcement learning algorithm on the initial prediction results, the labeled field values ​​and their true bounding box coordinates. During training, a weighted reward function for extraction and localization tasks is calculated; The reward value is calculated using the weighted reward function, and the parameters of the VLM model are updated using the GRPO algorithm to obtain a trained and optimized VLM model.

[0009] In one embodiment, the weighted reward function includes an extraction reward based on text editing distance normalization and a localization reward based on bounding box geometry; the localization reward is divided into a distance reward and a size reward: the distance reward is calculated based on the Euclidean distance between the center points of the predicted box and the ground truth box, and the size reward is calculated based on the area difference between the predicted box and the ground truth box.

[0010] In one embodiment, the VLM model training optimization step further includes: During training, a random offset is added to the predicted bounding box coordinates output by the training base model, and multiple variants are generated for each training data. Based on the number of fields, the multi-field extraction task is divided into single-field or multi-field sub-tasks of different difficulty levels. During training, the training data is progressively input into the model in the order from single-field to multi-field, thus constructing a course-style learning path from shallow to deep.

[0011] In one embodiment, the method further includes: A continuously optimized model iteration chain is constructed through a data feedback mechanism. The final localization result is pre-annotated by the trained and optimized VLM model and OCR combined service, and after manual review and correction, it is fed back to the training dataset for the next round of full or incremental training of the model.

[0012] In one embodiment, the steps for constructing the training dataset include: Obtain the original business dataset of document images; The data in the original business dataset is labeled to generate a training dataset containing the labeled field values ​​and their actual bounding box coordinates.

[0013] In one embodiment, the method further includes: In response to the instruction to reuse the VLM model, the reward function parameter configuration is adjusted according to the task requirements of the business scenario to reuse the training process of the VLM model.

[0014] Furthermore, to achieve the above objectives, this application also proposes a business document processing device based on a multimodal model, the device comprising: The acquisition module is used to acquire images of business documents; The coarse localization module is used to input the business document image into a pre-trained and optimized VLM model, and perform semantic extraction and coarse localization through the VLM model to obtain the semantic extraction result and the corresponding predicted bounding box as the coarse localization result. The VLM model is trained and optimized using the end-to-end GRPO reinforcement learning algorithm. The fine localization module is used to call the OCR model to refine the coarse localization result output by the VLM model and obtain the final localization result.

[0015] In addition, to achieve the above objectives, this application also proposes a business document processing device based on a multimodal model, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the business document processing method based on the multimodal model as described above.

[0016] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the business document processing method based on the multimodal model described above.

[0017] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the business document processing method based on a multimodal model as described above.

[0018] The present application proposes one or more technical solutions, which have at least the following technical effects: acquiring business document images; inputting the business document images into a pre-trained and optimized VLM model; performing semantic extraction and coarse localization through the VLM model to obtain semantic extraction results and corresponding predicted bounding boxes as coarse localization results; the VLM model is trained and optimized using an end-to-end GRPO reinforcement learning algorithm; and calling an OCR model to refine the coarse localization results output by the VLM model to obtain the final localization result. This application achieves an organic integration of VLM semantic understanding capabilities and OCR precise localization capabilities. VLM is responsible for understanding business semantics and providing field values ​​and approximate locations, while OCR is responsible for precise localization correction near the locations provided by VLM. This approach effectively solves the problem of inaccurate localization of the VLM model in scenarios with complex layouts and multiple repetitive fields, significantly improving the accuracy of field localization and the robustness of the recognition results. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the first embodiment of the business document processing method based on a multimodal model in this application. Figure 2 This is a schematic diagram of the module structure of the business document processing device based on a multimodal model according to an embodiment of this application; Figure 3 This is a schematic diagram of the hardware operating environment involved in the business document processing method based on a multimodal model in the embodiments of this application.

[0022] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0024] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0025] The main solution of this application embodiment is as follows: First, acquire a business document image. Then, input the business document image into a pre-trained and optimized VLM model. The VLM model performs semantic extraction and coarse localization to obtain the semantic extraction result and the corresponding predicted bounding box, which serves as the coarse localization result. The VLM model is trained and optimized using an end-to-end GRPO reinforcement learning algorithm. Finally, an OCR model is called to refine the coarse localization result output by the VLM model, resulting in the final localization result. This application achieves an organic integration of VLM semantic understanding capabilities and OCR precise localization capabilities. VLM is responsible for understanding business semantics and providing field values ​​and approximate locations, while OCR is responsible for precise localization correction near the location provided by VLM. This approach effectively solves the problem of inaccurate localization in scenarios with complex layouts and multiple repetitive fields, significantly improving the accuracy of field localization and the robustness of the recognition results.

[0026] This application's embodiments take into account that, in the field of cross-border business document review, traditional technologies mainly rely on OCR (Optical Character Recognition) technology combined with large language models to extract document information. OCR technology converts text in document images into editable text content through text recognition. However, traditional OCR technology has the following significant drawbacks: (1) Insufficient semantic understanding: Traditional OCR technology can only recognize the text content in images and cannot understand the semantic meaning of the text. In cross-border document review scenarios, documents contain a large number of professional terms, abbreviations and field names in specific formats (such as bill of lading number, waybill number, cargo identification, etc.). OCR technology can only mechanically recognize text symbols and cannot provide parsing results with a suitable reading order. As a result, the large language model cannot understand "which field represents what meaning" according to business needs, thus resulting in the extraction results lacking the support required by business semantics.

[0027] (2) Lack of semantic guidance in positioning: When multiple values ​​are repeated in a document image, traditional OCR positioning technology cannot determine which specific value the model is extracting. For example, in an export bill of lading, the text "CHINA" may appear in both the title and the table content. Business personnel expect to locate "CHINA" in the title, but traditional OCR positioning can only return all the identified "CHINA" locations and cannot intelligently filter according to semantic requirements, resulting in positioning results that do not match actual business needs.

[0028] (3) Poor adaptability to complex layouts: Cross-border documents often have complex layouts, including tables, columns, multi-column layouts and other complex structures. When faced with these complex layouts, traditional OCR technology is prone to problems such as text recognition errors and confusion in row and column correspondence, which further affects the accuracy of information extraction.

[0029] This application also considers that, with the development of Multimodal Large Model (VLM) technology, some studies have begun to attempt to use VLM for document information extraction. Compared with traditional OCR technology, VLM has a certain semantic understanding capability and can identify the meaning of fields based on prompts. However, existing VLM technology still has the following problems in practical applications: (1) Incompatibility between semantic understanding and localization ability: During the training process, the existing VLM model mainly optimizes the semantic understanding ability, while the training of the localization ability is insufficient. When faced with a joint task that requires the simultaneous output of extraction results and localization coordinates, although the model can understand the meaning of the field and output the correct text content, its localization results are often not accurate enough and cannot accurately pinpoint the specific location of the field in the image.

[0030] (2) Insufficient positioning accuracy in complex scenarios: In document images with complex structures and repetitive fields, the positioning capability of the VLM model is significantly insufficient. For example, when business personnel expect to locate "CHINA" in the title, the model may incorrectly take "CHINA" in the table below the title as the target, resulting in a positioning result that does not match the actual business needs. This problem is particularly prominent in scenarios with multiple repetitive fields.

[0031] (3) Lack of specific optimization for localization tasks: The reward function design of existing VLM models is mainly for semantic extraction tasks, without considering the special characteristics of localization tasks. Traditional localization evaluation metrics (such as IoU, Intersection over Union) often have a reward value of 0 when the model's initial localization ability is weak, which cannot provide effective training signals for the model, making it difficult to improve the model's localization ability.

[0032] To address the shortcomings of existing technologies, this invention provides a business document processing method based on a multimodal model (VLM), which can solve the following technical problems: the lack of coordination between semantic extraction and location positioning capabilities when the VLM model jointly performs these tasks; the discrepancy between positioning results and business semantic requirements in scenarios with multiple repeated fields; the cold start problem where the traditional IoU reward function cannot provide effective gradient guidance in the early stages of model training; and the need for further refinement and correction of the VLM model's positioning results. This achieves joint optimization of semantic understanding and positioning capabilities, improving the accuracy and robustness of document information extraction.

[0033] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device, business document processing device or system based on a multimodal model, etc., capable of realizing the above functions. The following description uses a business document processing system based on a multimodal model as an example to illustrate this embodiment and the subsequent embodiments.

[0034] To facilitate understanding of this invention, some terms are explained below: VLM (Vision-Language Model): An artificial intelligence model capable of processing both image and text modal information simultaneously, possessing visual understanding and language generation capabilities.

[0035] OCR (Optical Character Recognition): A technology that recognizes text content in an image and converts it into editable text.

[0036] GRPO (Group Relative Policy Optimization): A reinforcement learning algorithm that calculates the advantage function by normalizing the reward in the same batch of samples, without training an additional value network.

[0037] Bounding Box: A rectangular box used to mark the location of a target region in an image. It is usually represented by the pixel coordinates of the top left and bottom right corners in the format [x1, y1, x2, y2].

[0038] IoU (Intersection over Union): A metric that measures the degree of overlap between two bounding boxes, calculated as the ratio of the intersection area to the union area of ​​the two boxes.

[0039] Prompt: The instruction text input to the VLM model to guide the model in performing specific tasks.

[0040] Reinforcement learning is a machine learning method that learns optimal policies by having an agent interact with its environment to obtain reward signals.

[0041] Based on this, this application provides a business document processing method based on a multimodal model, specifically applied to the information extraction and location processing of cross-border business documents in a cross-border document review scenario. (Refer to...) Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the business document processing method based on a multimodal model according to this application.

[0042] In this embodiment, the business document processing method based on the multimodal model includes steps S10 to S30. The following provides a detailed explanation of each step.

[0043] like Figure 1 As shown, the first embodiment of this application proposes a business document processing method based on a multimodal model, the method comprising: Step S10: Obtain the image of the business document; Taking cross-border business scenarios as an example, business document images refer to electronic image files of various documents that require information extraction and verification in cross-border business scenarios, including but not limited to export bills of lading, import bills of lading, waybills, packing lists, invoices, customs declarations, and delivery orders. These document images can be obtained through scanners, document scanners, mobile phone photography, etc., and the image formats include but are not limited to JPEG, PNG, PDF, and TIFF.

[0044] After acquiring images of business documents, the system can perform preprocessing operations on the images, such as image denoising, tilt correction, and resolution normalization, to improve the accuracy and efficiency of subsequent processing.

[0045] In this embodiment, taking an export bill of lading as an example, the acquired image contains multiple business fields such as shipper information, consignee information, vessel name and voyage number, port of loading, port of destination, cargo description, and container number.

[0046] Step S20: Input the business document image into the pre-trained and optimized VLM model, and perform semantic extraction and coarse localization through the VLM model to obtain the semantic extraction result and the corresponding predicted bounding box as the coarse localization result. The VLM model is trained and optimized using the end-to-end GRPO reinforcement learning algorithm. Then, the business document image is input into a pre-trained and optimized VLM model. The VLM model is used to perform semantic extraction and coarse localization to obtain the semantic extraction result and the corresponding predicted bounding box, which are used as the coarse localization result.

[0047] Among them, VLM (Vision-Language Model) is an artificial intelligence model capable of processing both image and text modal information simultaneously. In this embodiment, the VLM model can be trained and optimized using Qwen3-VL-8B-Instruct as the base model. The Qwen3-VL-8B-Instruct model is a large multimodal model with a large number of parameters, supports mixed image and text input, and has powerful visual understanding and language generation capabilities.

[0048] In this embodiment, semantic extraction refers to the VLM model understanding the semantic meaning of each business field in the document image based on the input prompt and outputting the corresponding field values.

[0049] Coarse localization refers to the VLM model outputting the approximate location coordinates of a field in the image along with the semantic extraction result; this is known as the predicted bounding box. The predicted bounding box is typically represented in the format [x1, y1, x2, y2], where (x1, y1) are the pixel coordinates of the top-left corner of the bounding box in the image, and (x2, y2) are the pixel coordinates of the bottom-right corner of the bounding box. Coarse localization requires lower precision than precise localization; its purpose is to provide a general location range for subsequent OCR correction.

[0050] In this embodiment, the VLM model is constrained to output the semantic extraction results and the corresponding predicted bounding box coordinates in a predetermined JSON format.

[0051] In this embodiment, the VLM model is trained and optimized using the end-to-end GRPO reinforcement learning algorithm. GRPO (Group Relative Policy Optimization) is a reinforcement learning algorithm designed specifically for large language models and multimodal models. Unlike the traditional PPO (Proximal Policy Optimization) algorithm, GRPO does not require training a separate value function network (Critic network). Instead, it calculates the advantage function by normalizing the reward within a batch of samples.

[0052] Specifically, the GRPO algorithm, for the same input state (i.e., the same document image and the same prompt word), uses the current policy model to sample and generate multiple output results, calculates the reward value for each output result, and then uses the mean of these reward values ​​as a baseline to calculate the relative advantage of each output. This design makes GRPO more efficient when training large-scale models, reduces the need for additional value networks, and is particularly suitable for fine-tuning large reinforcement learning models such as VLM. In this embodiment, the specific process of VLM model training and optimization will be described in detail in subsequent embodiments.

[0053] In this embodiment, the business document image is input into a pre-trained and optimized VLM model. The VLM model performs semantic extraction and coarse localization to obtain the semantic extraction result and the corresponding predicted bounding box, which serves as the coarse localization result. The coarse localization result refers to the complete output of the VLM model, which includes the semantic extraction result and the coordinates of the predicted bounding box. This result, as an intermediate product, is passed to the next step for fine-tuning.

[0054] Step S30: Call the OCR model to refine the coarse localization result output by the VLM model to obtain the final localization result.

[0055] Then, the OCR model is called to refine the coarse localization results output by the VLM model, and the final localization result is obtained.

[0056] OCR (Optical Character Recognition) is a technology specifically designed to recognize text content in images and output the text along with its location coordinates. Unlike VLM models, OCR models focus on text detection and recognition tasks, and can output the location and corresponding text content of each text region in an image with high accuracy. Commonly used OCR models include PaddleOCR, Tesseract, and Google Cloud Vision OCR.

[0057] In this embodiment, fine-grained correction refers to using the high-precision positioning capabilities of the OCR model to correct and optimize the coarse positioning results output by the VLM model. The core idea is that the VLM model excels at semantic understanding, accurately determining "which field value corresponds to which business meaning," but its positioning accuracy is limited; the OCR model excels at precise positioning, accurately providing the precise coordinates of each text region in the image, but it lacks semantic understanding capabilities. By combining the advantages of both, a synergistic effect of "VLM defining semantics and OCR defining location" can be achieved.

[0058] Specifically, in this embodiment, the OCR fine-tuning correction includes the following process: First, the OCR model is invoked to perform full text detection and recognition on the business document images obtained in the above steps. This involves detecting the location of all text regions in the image and recognizing the text content in each region. The OCR model outputs a list of text boxes, each containing text content and its precise bounding box coordinates.

[0059] Secondly, using the field values ​​in the semantic extraction results output by the VLM model as the matching key, text boxes with matching text content are selected from all text boxes recognized by OCR, forming a candidate text box set.

[0060] Then, the geometric distance between each text box in the candidate text box set and the predicted bounding box output by the VLM model is calculated. The geometric distance can be measured using the Euclidean distance between the center points of the text boxes.

[0061] Finally, the text box with the smallest geometric distance is selected, and its precise coordinates are used to replace the predicted bounding box coordinates output by the VLM model, as the final localization result.

[0062] The above steps specifically involve: acquiring business document images; inputting these images into a pre-trained and optimized VLM model; performing semantic extraction and coarse localization using the VLM model to obtain semantic extraction results and corresponding predicted bounding boxes as coarse localization results; and finally, using an OCR model to refine the coarse localization results output by the VLM model to obtain the final localization result. This embodiment organically integrates the semantic understanding capabilities of VLM with the precise localization capabilities of OCR: VLM is responsible for understanding business semantics and providing field values ​​and approximate locations, while OCR is responsible for precise localization correction near the locations provided by VLM. This approach effectively solves the problem of inaccurate localization of VLM models in scenarios with complex layouts and multiple repetitive fields, significantly improving the accuracy of field localization and the robustness of recognition results.

[0063] Based on the first embodiment described above, a second embodiment of this application is proposed. In this embodiment, the coarse localization result output by the VLM model is finely corrected by calling the OCR model, and the final localization result is further refined.

[0064] As one implementation method, step S30 involves calling an OCR model to refine the coarse localization result output by the VLM model, obtaining the final localization result, including: Step A1: Call the OCR model to perform full text detection and recognition on the business document image to obtain multiple text boxes and their corresponding text content and precise coordinates; Step A2: Using the field values ​​in the semantic extraction results as the matching key, filter out a set of candidate text boxes with consistent text content from the text boxes; Step A3: Calculate the geometric distance between each text box in the candidate text box set and the predicted bounding box; Step A4: Select the text box with the smallest geometric distance, and replace the predicted bounding box with the precise coordinates of the text box with the smallest geometric distance as the final positioning result output.

[0065] Specifically, firstly, the OCR model is invoked to perform full text detection and recognition on the business document image, obtaining multiple text boxes and their corresponding text content and precise coordinates.

[0066] Full text detection refers to the OCR model performing a comprehensive scan and analysis of the input business document image to detect all regions containing text. Text detection typically employs deep learning-based object detection algorithms (such as DBnet, EAST, PSENet, etc.), which can output the bounding box coordinates of each text region.

[0067] Text recognition refers to the process of converting text symbols in an image into an editable text string by using sequence recognition algorithms (such as CRNN, Attention-based Recognition, etc.) for each detected text region.

[0068] In this embodiment, each text box output by the OCR model contains three elements: text content (i.e., the recognized text string), precise coordinates (i.e., the bounding box position of the text box in the image, represented in the format [x1, y1, x2, y2]), and confidence score (representing the credibility of the recognition result).

[0069] The purpose of full text detection and recognition is to provide a complete candidate data pool for subsequent matching and filtering. Because the OCR model detects and recognizes all text regions, the corresponding text box can be found in the OCR result no matter where the field value output by the VLM model appears in the image.

[0070] Then, using the field values ​​in the semantic extraction results as the matching key, a set of candidate text boxes with consistent text content is filtered out from the text boxes.

[0071] A matching key is a key field value used for searching and matching within a dataset. In this step, the matching key is a field value from the semantic extraction results output by the VLM model.

[0072] The selection process for the candidate text box set is as follows: All text boxes output by the OCR model are traversed, and the text content of each text box is compared with the matching key. If the text content is exactly the same as the matching key (or matches within a preset edit distance threshold), the text box is included in the candidate text box set. During string comparison, case normalization (e.g., converting all letters to uppercase or lowercase) and special character processing (e.g., removing spaces, punctuation marks, etc.) can be performed to improve the robustness of the matching.

[0073] For example, if the VLM model outputs the field value "SHANGHAI", and the OCR model identifies two text boxes containing "SHANGHAI" during full detection (one in the title area and one in the table content area), then both text boxes will be included in the candidate text box set.

[0074] The purpose of this step is to use the semantic understanding capability of the VLM model to determine "what value to look for", and then use the comprehensive detection capability of the OCR model to find "where this value appears", thereby narrowing the localization problem from "blindly searching in the whole image" to "selecting from several candidate locations".

[0075] Then, the geometric distance between each text box in the candidate text box set and the predicted bounding box is calculated.

[0076] Geometric distance is a metric that measures how close two bounding boxes are in spatial location. In this embodiment, geometric distance is calculated using the Euclidean distance between the center points of the two bounding boxes. The smaller the geometric distance, the closer the candidate text box is in spatial location to the location predicted by the VLM model, and therefore the more likely it is to be the target text box that the VLM model originally intended to locate.

[0077] This step aims to further filter and rank candidate text boxes using spatial location information. When multiple text boxes contain identical content, geometric distance can help determine which one best matches the VLM semantic understanding result.

[0078] Then, the text box with the smallest geometric distance is selected, and the predicted bounding box is replaced with the precise coordinates of the text box with the smallest geometric distance as the final positioning result.

[0079] In this process, the text box with the smallest geometric distance is selected, meaning the text box that is spatially closest to the location predicted by the VLM model is chosen as the final localization result. Then, the precise coordinates of this text box are output as the final localization coordinates of the field.

[0080] The final localization result consists of two parts: semantic extraction results (from the VLM model) and precise localization coordinates (from the OCR model).

[0081] Through the above steps, this embodiment achieves a deep integration of VLM semantic understanding capabilities and OCR precise positioning capabilities. VLM is responsible for semantic understanding (knowing which field values ​​to extract), while OCR is responsible for precise positioning (knowing the precise location of the value in the image). This two-stage fusion strategy of semantic guidance and geometric filtering effectively solves the dual problems of insufficient positioning accuracy of VLM and lack of semantic understanding in traditional methods, and is particularly suitable for scenarios with repetitive fields and complex layouts in cross-border documents. After the correction processing in this embodiment, the accuracy of field positioning is significantly improved, providing reliable location information support for subsequent business review and data processing.

[0082] Based on any of the above embodiments of this application, a third embodiment of this application is proposed. In this embodiment, a detailed description is given of how to use the end-to-end GRPO reinforcement learning algorithm to train and optimize the VLM model so that it has the ability to perform semantic extraction and coarse localization simultaneously.

[0083] As one implementation method, the VLM model training and optimization steps include: Step B1: Obtain the training dataset and the training base model. The training dataset is constructed based on the original document image data containing labeled field values ​​and their true bounding box coordinates. Step B2: Based on the training dataset, input the prompt words containing the fixed extraction instruction template and the field variables to be extracted into the training base model, and constrain the training base model to output the semantic extraction results of the field and the corresponding predicted bounding box coordinates in a predetermined JSON format to obtain the initial prediction results. Step B3: Using the initial prediction results, the labeled field values, and their true bounding box coordinates, the VLM model is trained with all parameters using the end-to-end GRPO reinforcement learning algorithm. Step B4: Calculate the weighted reward function for the extraction and localization tasks during training; Step B5: Calculate the reward value using the weighted reward function and update the parameters of the VLM model using the GRPO algorithm to obtain the trained and optimized VLM model.

[0084] First, obtain the training dataset and the training base model.

[0085] The training dataset was constructed based on original document image data containing labeled field values ​​and their actual bounding box coordinates.

[0086] The construction scheme for the training dataset can include: First, obtaining the original business dataset of document images. The original business dataset comes from real cross-border business scenarios, including images of various types of business documents (export bills of lading, import bills of lading, waybills, invoices, etc.), covering different layout designs, languages, image qualities, and shooting conditions. Second, annotating the data in the original business dataset to generate a training dataset containing annotated field values ​​and their ground truth bounding box coordinates. The annotation work is completed by professional annotators, and the annotation content includes: for each business field that needs to be extracted from each document image, the annotators need to annotate the text value of the field (i.e., the field value) and the precise location of the text in the image (i.e., the ground truth bounding box coordinates, represented in the format [x1, y1, x2, y2]).

[0087] The training base model refers to the original VLM model that has not been optimized in this training. In this embodiment, Qwen3-VL-8B-Instruct is selected as the training base model. This model is an open-source 8 billion-parameter multimodal large model from the Alibaba Cloud Qwen team, supporting mixed input of images, text, and video, and possessing powerful visual understanding and language generation capabilities. The reason for choosing this model as the base is that it has a good foundation in multimodal understanding, can process image and text input simultaneously, and is suitable as a base model for document information extraction tasks.

[0088] Then, based on the training dataset, prompt words containing a fixed extraction instruction template and the field variables to be extracted are input into the training base model. The training base model is constrained to output the semantic extraction results of the field and the corresponding predicted bounding box coordinates in a predetermined JSON format to obtain the initial prediction results.

[0089] The prompt is the instruction text input to the VLM model, which guides the model to perform specific tasks.

[0090] In this embodiment, the prompt words adopt a modular design, divided into constant parts and variable parts: The constant part refers to the fixed extraction instruction template, which includes unchanging content such as task description and output format requirements.

[0091] The variable part refers to the knowledge of the fields to be extracted, that is, which business fields need to be extracted in this task. These fields change dynamically according to different business scenarios and document types.

[0092] By decoupling the constant and variable parts, modularity and reusability of prompt words are achieved. When faced with different document types or different extraction requirements, only the variable part needs to be modified, without redesigning the entire prompt word template.

[0093] In this context, constraining the training daemon model to output in a predetermined JSON format means forcing the model to output results in a uniform format through explicit instructions in the prompts and / or format constraints during model decoding. A uniform JSON format ensures the structured nature of the output, facilitating subsequent parsing and processing.

[0094] The initial prediction results refer to the predictions made by the training base model based on the current prompt word and input image without optimization during this training phase. These initial prediction results typically suffer from inaccurate localization and semantic understanding biases, requiring optimization through subsequent training steps.

[0095] Then, the VLM model is trained with all parameters using the end-to-end GRPO reinforcement learning algorithm, based on the initial prediction results, the labeled field values, and their true bounding box coordinates.

[0096] End-to-end training refers to the complete training process from the original input to the final output, without the need for stages.

[0097] In this embodiment, end-to-end training means that the model learns the semantic extraction results and the output of the location coordinates directly from the document image and prompt words input, without needing to go through independent feature extraction, text recognition and other intermediate steps.

[0098] Full parameter training refers to updating all parameters of the model during the training process, rather than updating only some parameters (such as updating only the output layer or only a few adapter parameters). Full parameter training can make full use of the training data and optimize all parts of the model, but it requires more computing resources. In this embodiment, a single-machine 8-GPU 910B training environment can be used, and the vllm-ascend0.11.0 image can be used to complete the dependency setup.

[0099] GRPO (Group Relative Policy Optimization) is a reinforcement learning algorithm. In this embodiment, the specific workflow of the GRPO algorithm during training is as follows: For each document image and corresponding prompt word (i.e., a "state") in the training dataset, the GRPO algorithm samples and generates multiple distinct outputs using the current policy model (i.e., the VLM model being trained). Each output contains semantic extraction results and predicted bounding box coordinates. Then, a reward value is calculated for each output (the calculation of the reward function is detailed in the next step). Next, using the mean of the reward values ​​of this set of outputs as a baseline, the relative advantage of each output is calculated. Finally, the model parameters are updated using policy gradients based on the advantage value, increasing the probability of generating high-reward outputs and decreasing the probability of generating low-reward outputs.

[0100] The main difference between GRPO and the traditional PPO algorithm is that PPO requires training an additional value network (Critic network) to estimate state values, while GRPO directly uses the average reward of multiple sampled outputs within the same batch as a baseline, eliminating the need to train an additional value network. This makes GRPO more efficient when training large-scale VLM models, reducing memory usage and computational overhead.

[0101] Then, during training, a weighted reward function for the extraction and localization tasks is calculated.

[0102] The reward function is a core component in reinforcement learning, used to evaluate the quality of the model's output and guide the direction of model parameter updates. In this embodiment, the reward function is designed for both extraction and localization tasks, employing a weighted combination.

[0103] The weighted reward function includes an extraction reward based on text edit distance normalization and a localization reward based on bounding box geometry.

[0104] The extraction reward is used to evaluate the accuracy of the model's semantic extraction. In this embodiment, the extraction reward is calculated based on OCR edit distance normalization. OCR edit distance (Levenshtein distance) refers to the minimum number of editing operations (including insertion, deletion, and replacement of characters) required to convert one string into another. The OCR edit distance between the model's output field value and the labeled true field value is normalized (e.g., divided by the maximum length of the field value) to obtain a normalized distance value between 0 and 1. This value is then subtracted from 1 to obtain the extraction reward. The smaller the edit distance, the closer the model's output field value is to the true value, and the higher the extraction reward.

[0105] In this embodiment, the positioning reward is divided into a distance reward and a size reward: The distance reward is calculated based on the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box. Specifically, the Euclidean distance between the center points of the predicted and ground truth bounding boxes is calculated and then linearly normalized using the diagonal length of the ground truth bounding box. The formula is: Distance Reward = 1 - (Center Point Distance / Ground Truth Bounding Box Diagonal Length). The closer the center points are, the closer the distance reward is to 1.

[0106] Size reward is calculated based on the area difference between the predicted bounding box and the ground truth bounding box. Specifically, the relative difference between the predicted and ground truth bounding box areas is calculated (i.e., the absolute value of the area difference divided by the ground truth bounding box area), and then normalized using a preset area difference tolerance threshold. The calculation formula is: Size reward = 1 - (relative area difference / threshold). The smaller the area difference, the closer the size reward is to 1.

[0107] Then, the distance reward and size reward are multiplied by their respective weights and summed to obtain the final location reward. For example: Location reward = α × distance reward + β × size reward, where α and β are preset weight coefficients.

[0108] Ultimately, the total reward = γ × extraction reward + δ × localization reward, where γ and δ are preset weighting coefficients used to balance the importance of extraction and localization tasks during training.

[0109] The innovation of the reward function design in this step lies in splitting the traditional IoU reward function into two independent dimensions: distance reward and size reward. The traditional IoU metric has a value of 0 when the predicted bounding box and the ground truth bounding box have no intersection, resulting in the complete disappearance of the reward signal and preventing the model from obtaining effective gradient guidance in the early stages of training. However, the distance and size rewards in this embodiment can still provide non-zero reward signals even when the predicted bounding box and the ground truth bounding box do not overlap. As long as the distance between the center points or the area difference is finite, the model can find an optimization direction. This allows the model to find a direction for improvement even if the localization performance is poor in the early stages of training, effectively solving the cold start problem.

[0110] Then, the reward value is calculated using the weighted reward function and the parameters of the VLM model are updated using the GRPO algorithm to obtain the trained and optimized VLM model.

[0111] Specifically, in each training iteration, for each sampled output, its reward value is calculated using the weighted reward function defined above. Then, the GRPO algorithm calculates the advantage value for each output, using the mean reward of multiple outputs within the same batch as a baseline. Finally, the parameters of the VLM model are updated based on the advantage value using the policy gradient.

[0112] In addition, this embodiment also sets a KL divergence constraint coefficient (kl_loss_coef=0.001) during the training process to constrain the magnitude of model parameter updates, prevent the model from deviating too far from the original distribution, and maintain the stability of training.

[0113] In one implementation method, the main training parameters are set as follows: global training batch size train_batch_size is 64, PPO micro-batch size ppo_mini_batch_size is 16, log probability micro-batch size log_prob_micro_batch_size_per_gpu is 4, number of samples rollout_n is 4, learning rate lr is 5e-7, and training steps training_steps is 300.

[0114] Furthermore, in order to prevent the model from overfitting to the coordinate data, this embodiment can also add random offsets to the predicted bounding box coordinates output by the training base model during the training process, generating multiple variants for each training data.

[0115] Here, random offset refers to a small random perturbation added to the coordinate labels of the training data. In this embodiment, a random offset within the range of 0 to 5 pixels is added to the labeled coordinates [x1, y1, x2, y2] in the training data, generating three variant data for each original data.

[0116] For example, if the original labeled coordinates are [118, 43, 282, 77], adding a random offset may generate variants such as [120, 45, 280, 75], [116, 41, 284, 79], and [119, 44, 281, 76].

[0117] The purpose of this step is to prevent the model from overfitting to the coordinate data. If the coordinates in the training data are all precise labeled values, the model may "memorize" these specific coordinate values, resulting in poor performance when faced with images in real-world applications where the coordinates have slight deviations. By adding random offsets, the model learns the ability to generalize and locate the correct region even when the coordinates fluctuate within a certain range, thereby improving the model's robustness.

[0118] Furthermore, as an implementation method, the multi-field extraction task can be divided into single-field or multi-field sub-tasks of different difficulty levels according to the number of fields. During training, the training data is progressively input into the model in the order from single-field to multi-field, thus constructing a course-based learning path from shallow to deep.

[0119] In this embodiment, the original multi-field joint extraction task is broken down into sub-tasks of different difficulty levels, such as single-field task (extracting only 1 field), double-field task (extracting 2 fields at the same time), and triple-field task (extracting 3 fields at the same time).

[0120] During training, the model is first trained using single-field task data, allowing it to learn the basic ability to find and locate a specific field in an image. Then, two-field and three-field tasks are gradually introduced, allowing the model to learn the complex ability to process multiple fields simultaneously and distinguish the positions of different fields. Finally, the model is trained using complete full-field task data.

[0121] The purpose of this step is to reduce the learning difficulty for the model in complex multi-field tasks. If the model is initially trained to extract and locate multiple fields simultaneously, it may struggle to optimize multiple output targets at the same time. Through progressive training from simple to complex, the model can first master basic capabilities and then gradually add more complex capabilities, ultimately achieving good performance on multi-field joint extraction tasks.

[0122] Through the above steps, the VLM model is trained and optimized using the end-to-end GRPO reinforcement learning algorithm, eliminating the need for the traditional two-stage training process of SFT (supervised fine-tuning) + GRPO. This design simplifies the training process, reduces the resource investment in constructing complete model response data, and avoids the overfitting problem that may occur in the SFT stage under long-word prompt tasks, making training more efficient and stable. After training optimization in this embodiment, the model shows significant improvements in both semantic understanding and position awareness.

[0123] Based on any of the above embodiments, a fourth embodiment of this application is proposed. In this embodiment, the method further includes constructing a continuously optimized model iteration chain through a data feedback mechanism. Specifically, the final localization result is pre-annotated using the trained and optimized VLM model combined with an OCR service, and after manual review and correction, it is fed back to the training dataset for the next round of full or incremental model training.

[0124] Specifically, data backflow refers to feeding back the data and results generated by the model in actual business applications to the training dataset for subsequent model optimization and iteration. The core idea of ​​the data backflow mechanism is to build a closed-loop iterative chain of "inference → pre-labeling → manual correction → retraining".

[0125] Specifically, in actual business scenarios, whenever the system processes a business document image and outputs the final location result, the document image and its processing result are not discarded, but collected for subsequent model optimization.

[0126] Pre-annotation refers to the automatic generation of annotation results for newly collected document images using existing automated tools (in this embodiment, a combination service of a trained and optimized VLM model and an OCR model). Specifically, the combined service processes the document images according to the method flow described in the first embodiment (VLM semantic extraction and coarse localization, OCR fine-tuning correction) to generate pre-annotation results containing field values ​​and precise coordinates.

[0127] Pre-annotation is far more efficient than manual annotation, but its accuracy may be slightly lower. The purpose of pre-annotation is to significantly reduce the workload of manual annotation. Manual annotators only need to review and correct the pre-annotated results, instead of annotating word by word from scratch.

[0128] After manual review and correction, the data is fed back into the training dataset. Specifically, after the pre-annotated results are generated, they are manually reviewed and corrected by professional annotators. The review includes checking whether the field values ​​extracted by the VLM model are correct (if incorrect, they are corrected to the correct values), and checking whether the location coordinates after OCR correction are accurate (if inaccurate, the bounding box positions are adjusted). Manual review and correction ensure the quality of the fed-back data and prevent erroneous data from being introduced into the training set.

[0129] The data, after being manually reviewed and corrected, along with its corresponding document images, is added back to the training dataset to become training samples for the next round of model training, used for full or incremental training of the next round of model.

[0130] Once a certain amount of backflow data has accumulated, a new round of model training is triggered. Training can be performed using full training (retraining the model using all training data) or incremental training (fine-tuning the existing model using new data).

[0131] Full training refers to training a model from scratch using all available training data (including original training data and all reflow data). Full training makes full use of all data, but it is computationally expensive.

[0132] Incremental training refers to further training and optimization based on the current model using only newly added backflow data. Incremental training has lower computational costs and is suitable for frequent model updates.

[0133] Through the above steps, a complete iterative chain is constructed, from pre-labeling of combined capability services → manual correction → data feedback into the database → model retraining. As the business system's runtime increases, the feedback data accumulates continuously, allowing the model to continuously evolve using this data from real business scenarios. Its performance can automatically and continuously improve without additional manual data collection costs. This data-driven positive feedback loop mechanism enables the technical solution of this invention to have continuous self-optimization capabilities, adapting to changes in business scenarios and new document types, thus possessing extremely high practical and commercial value.

[0134] Based on any of the above embodiments, a fifth embodiment of this application is proposed. In this embodiment, the method further includes adjusting the reward function parameter configuration according to the task requirements of the business scenario to reuse the training process of the VLM model in response to the reuse instruction of the VLM model.

[0135] Here, reuse instructions refer to instructions issued by users or the system to use existing VLM model training processes to handle new tasks or scenarios. The core of this embodiment lies in constructing a reusable multimodal reinforcement learning training paradigm, enabling the training process to be quickly reused and adapted in different business scenarios.

[0136] Specifically, when faced with a new business scenario (such as switching from export bill of lading processing to import customs declaration processing, from cross-border logistics documents to cross-border e-commerce orders, from English documents to multilingual documents, etc.) or a new task requirement, users can initiate the application of the training process of this application embodiment in the new scenario by sending a reuse instruction.

[0137] The reuse instruction can take the form of an API call, a configuration file, or a command-line argument. For example, users can instruct the system to switch to a new task scenario by modifying parameters such as task type, field list, and data path in the configuration file.

[0138] The system adjusts the reward function parameter configuration according to the task requirements of the business scenario to reuse the training process of the VLM model. The reward function parameter configuration refers to the various weights and parameters in the weighted reward function described in the above steps, including but not limited to: the weight coefficients (γ and δ) for extraction reward and location reward, the weight coefficients (α and β) for distance reward and size reward, the area difference tolerance threshold in the size reward, and the edit distance normalization parameter in the extraction reward.

[0139] Different business scenarios may have different requirements for extraction accuracy and positioning accuracy. For example, in scenarios requiring high-precision positioning (such as needing to accurately select field locations for manual verification), the weight δ of the positioning reward can be increased to make the model focus more on positioning accuracy; in scenarios where field values ​​are easily confused (such as having a large number of similar texts), the weight γ of the extraction reward can be increased to make the model focus more on semantic extraction accuracy; in scenarios with high-quality document images and clear text box boundaries, the weight β of the size reward can be reduced or the area difference tolerance threshold can be increased to reduce excessive penalties for size accuracy; in scenarios with low-quality document images and blurred text box boundaries, the weight β of the size reward can be increased or the area difference tolerance threshold can be reduced to strengthen the constraints on size accuracy.

[0140] By adjusting these parameter configurations, the training process can be adapted to different business scenarios and task requirements without modifying the core code and algorithm logic of the training process.

[0141] After adjusting the reward function parameters, the training process can be started using the new parameter configuration to optimize the VLM model for the new scenario. Other steps in the training process (data preparation, prompt word design, GRPO training, OCR correction, etc.) remain unchanged.

[0142] Through the above steps, this embodiment constructs a reusable multimodal reinforcement learning training paradigm, including data construction and expansion strategies, a general reward function framework, etc. When facing similar tasks (such as invoice recognition, contract review, form understanding, etc.), the team only needs to adjust the reward function parameter configuration according to its own task requirements to quickly reuse this training process, greatly reducing the threshold and cost of model optimization. This reusability and scalability make the technical solution of this invention not only applicable to the specific scenario of cross-border document processing, but also extendable to a wider range of document intelligent processing fields, with broad application prospects.

[0143] Compared with the prior art, the technical solution of this embodiment has the following advantages: 1. Simplified training process and improved efficiency. This application adopts an end-to-end GRPO reinforcement learning method, which eliminates the need for the traditional two-stage SFT+GRPO training, simplifies the training process, reduces resource investment, and avoids overfitting problems that may be caused by the SFT stage, making training more efficient and stable.

[0144] 2. Innovative reward function design with strong training guidance. This application splits the localization reward into two dimensions: distance reward and size reward. This solves the problem that the traditional IoU metric has a reward of 0 when the localization offset occurs, allowing the model to find optimization directions even when the initial localization performance is poor, thus providing stronger training guidance.

[0145] 3. Deep integration of multimodal model semantic capabilities and OCR positioning capabilities. This application is the first to achieve the organic integration of VLM semantic understanding capabilities and OCR precise positioning capabilities in a document processing scenario. VLM is responsible for semantic extraction and coarse positioning, while OCR is responsible for fine-grained correction, effectively solving the problem of inaccurate positioning by VLM in scenarios with complex layouts and repetitive fields.

[0146] 4. Joint optimization of extraction and localization for improved overall performance. This application designs a weighted reward function for extraction and localization tasks, enabling the model to simultaneously focus on the accuracy of field recognition and the rationality of location positioning during training.

[0147] 5. The solution is highly reusable and scalable. This application constructs a reusable multimodal reinforcement learning training paradigm. When facing similar tasks in the future, only the reward function parameters need to be adjusted for rapid reuse, which greatly reduces the threshold and cost of model optimization.

[0148] Furthermore, to achieve the above objectives, this application also proposes a business document processing device based on a multimodal model, such as... Figure 2 As shown, the device includes: Module 10 is used to acquire images of business documents; The coarse localization module 20 is used to input the business document image into a pre-trained and optimized VLM model, and perform semantic extraction and coarse localization through the VLM model to obtain the semantic extraction result and the corresponding predicted bounding box as the coarse localization result. The VLM model is trained and optimized using the end-to-end GRPO reinforcement learning algorithm. The fine positioning module 30 is used to call the OCR model to refine the coarse positioning result output by the VLM model and obtain the final positioning result.

[0149] The business document processing device based on a multimodal model provided in this application adopts the business document processing method based on a multimodal model in the above embodiments, realizing the joint optimization of semantic understanding and localization capabilities, and improving the accuracy and robustness of document information extraction. Compared with the prior art, the beneficial effects of the business document processing device based on a multimodal model provided in this application are the same as those of the business document processing method based on a multimodal model provided in the above embodiments, and other technical features in the business document processing device based on a multimodal model are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0150] In addition, to achieve the above objectives, this application also proposes a business document processing device based on a multimodal model, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the business document processing method based on the multimodal model as described above.

[0151] The following is for reference. Figure 3 This document illustrates a structural diagram of a business document processing device based on a multimodal model suitable for implementing embodiments of this application. The business document processing device based on a multimodal model in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 3 The business document processing device based on the multimodal model shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0152] like Figure 3As shown, a business document processing device based on a multimodal model may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the business document processing device based on the multimodal model. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the multimodal model-based business document processing equipment to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows a multimodal model-based business document processing equipment with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0153] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0154] The business document processing device based on a multimodal model provided in this application adopts the business document processing method based on a multimodal model in the above embodiments, realizing the joint optimization of semantic understanding and localization capabilities, and improving the accuracy and robustness of document information extraction. Compared with the prior art, the beneficial effects of the business document processing device based on a multimodal model provided in this application are the same as those of the business document processing method based on a multimodal model provided in the above embodiments, and other technical features in this business document processing device based on a multimodal model are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0155] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0156] In addition, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the business document processing method based on the multimodal model described above.

[0157] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0158] The aforementioned computer-readable storage medium may be included in a business document processing device based on a multimodal model; or it may exist independently and not be assembled into a business document processing device based on a multimodal model.

[0159] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a business document processing device based on a multimodal model, the multimodal model-based business document processing device performs the following: acquires a business document image; inputs the business document image into a pre-trained and optimized VLM model; performs semantic extraction and coarse localization using the VLM model to obtain semantic extraction results and corresponding predicted bounding boxes as coarse localization results; the VLM model is trained and optimized using an end-to-end GRPO reinforcement learning algorithm; and calls an OCR model to refine the coarse localization results output by the VLM model to obtain the final localization result.

[0160] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0161] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0162] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0163] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described business document processing method based on a multimodal model. This achieves joint optimization of semantic understanding and localization capabilities, improving the accuracy and robustness of document information extraction. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the business document processing method based on a multimodal model provided in the above embodiments, and will not be repeated here.

[0164] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the business document processing method based on a multimodal model as described above.

[0165] The computer program product provided in this application achieves joint optimization of semantic understanding and localization capabilities, improving the accuracy and robustness of document information extraction. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the business document processing method based on a multimodal model provided in the above embodiments, and will not be repeated here.

[0166] The present application proposes one or more technical solutions, which have at least the following technical effects: acquiring business document images; inputting the business document images into a pre-trained and optimized VLM model; performing semantic extraction and coarse localization through the VLM model to obtain semantic extraction results and corresponding predicted bounding boxes as coarse localization results; the VLM model is trained and optimized using an end-to-end GRPO reinforcement learning algorithm; and calling an OCR model to refine the coarse localization results output by the VLM model to obtain the final localization result. This application achieves an organic integration of VLM semantic understanding capabilities and OCR precise localization capabilities. VLM is responsible for understanding business semantics and providing field values ​​and approximate locations, while OCR is responsible for precise localization correction near the locations provided by VLM. This approach effectively solves the problem of inaccurate localization of the VLM model in scenarios with complex layouts and multiple repetitive fields, significantly improving the accuracy of field localization and the robustness of the recognition results.

[0167] It should be noted that the business document processing method, system, equipment, and storage medium based on a multimodal model provided in this application can be widely applied to automated document processing systems in fields such as cross-border logistics, international trade, customs declaration, and financial document review. Through the technical solution of this application, efficient and accurate information extraction and location of various business documents can be achieved, significantly reducing the workload of manual data entry and review, improving business processing efficiency and accuracy, and possessing significant industrial practical value and economic benefits.

[0168] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A business document processing method based on a multimodal model, characterized in that, The method includes: Obtain images of business documents; The business document image is input into a pre-trained and optimized VLM model. The VLM model is used to perform semantic extraction and coarse localization to obtain the semantic extraction result and the corresponding predicted bounding box as the coarse localization result. The VLM model is trained and optimized using the end-to-end GRPO reinforcement learning algorithm. The coarse localization result output by the VLM model is finely corrected by calling the OCR model to obtain the final localization result.

2. The method as described in claim 1, characterized in that, The process of calling the OCR model to refine the coarse localization result output by the VLM model yields the final localization result, including: The OCR model is invoked to perform full text detection and recognition on the business document image, and multiple text boxes and their corresponding text content and precise coordinates are obtained. Using the field values ​​in the semantic extraction results as the matching key, a set of candidate text boxes with consistent text content is filtered out from the text boxes. Calculate the geometric distance between each text box in the candidate text box set and the predicted bounding box; The text box with the smallest geometric distance is selected, and its precise coordinates are used to replace the predicted bounding box as the final positioning result.

3. The method as described in claim 1, characterized in that, The steps for training and optimizing the VLM model include: Obtain a training dataset and a training base model. The training dataset is constructed based on original document image data containing labeled field values ​​and their true bounding box coordinates. Based on the training dataset, prompt words containing a fixed extraction instruction template and field variables to be extracted are input into the training base model. The training base model is constrained to output the semantic extraction results of the field and the corresponding predicted bounding box coordinates in a predetermined JSON format to obtain the initial prediction results. The VLM model is trained with all parameters using the end-to-end GRPO reinforcement learning algorithm on the initial prediction results, the labeled field values ​​and their true bounding box coordinates. During training, a weighted reward function for extraction and localization tasks is calculated; The reward value is calculated using the weighted reward function, and the parameters of the VLM model are updated using the GRPO algorithm to obtain a trained and optimized VLM model.

4. The method as described in claim 3, characterized in that, The weighted reward function includes an extraction reward based on text editing distance normalization and a localization reward based on bounding box geometry. The localization reward is divided into a distance reward and a size reward: the distance reward is calculated based on the Euclidean distance between the center points of the predicted box and the ground truth box, and the size reward is calculated based on the area difference between the predicted box and the ground truth box.

5. The method as described in claim 4, characterized in that, The VLM model training and optimization steps also include: During training, a random offset is added to the predicted bounding box coordinates output by the training base model, and multiple variants are generated for each training data. Based on the number of fields, the multi-field extraction task is divided into single-field or multi-field sub-tasks of different difficulty levels. During training, the training data is progressively input into the model in the order from single-field to multi-field, thus constructing a course-style learning path from shallow to deep.

6. The method as described in claim 3, characterized in that, The method further includes: A continuously optimized model iteration chain is constructed through a data feedback mechanism. The final localization result is pre-annotated by the trained and optimized VLM model and OCR combined service, and after manual review and correction, it is fed back to the training dataset for the next round of full or incremental training of the model.

7. The method as described in claim 3, characterized in that, The steps for constructing the training dataset include: Obtain the original business dataset of document images; The data in the original business dataset is labeled to generate a training dataset containing the labeled field values ​​and their actual bounding box coordinates.

8. The method as described in claim 2, characterized in that, The method further includes: In response to the instruction to reuse the VLM model, the reward function parameter configuration is adjusted according to the task requirements of the business scenario to reuse the training process of the VLM model.

9. A business document processing device based on a multimodal model, characterized in that, The device includes: The acquisition module is used to acquire images of business documents; The coarse localization module is used to input the business document image into a pre-trained and optimized VLM model, and perform semantic extraction and coarse localization through the VLM model to obtain the semantic extraction result and the corresponding predicted bounding box as the coarse localization result. The VLM model is trained and optimized using the end-to-end GRPO reinforcement learning algorithm. The fine localization module is used to call the OCR model to refine the coarse localization result output by the VLM model and obtain the final localization result.

10. A business document processing device based on a multimodal model, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the business document processing method based on a multimodal model as described in any one of claims 1 to 8.

11. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the business document processing method based on a multimodal model as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the business document processing method based on a multimodal model as described in any one of claims 1 to 8.

Images