Optical character recognition method and system based on multi-mode large model
By using reinforcement learning and data augmentation training on a multimodal large model, the shortcomings of traditional OCR systems in structured document image processing are addressed, improving the accuracy of structured information extraction and semantic alignment capabilities, and achieving efficient and accurate document understanding.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-11
- Publication Date
- 2026-03-03
AI Technical Summary
Traditional OCR systems are inadequate when processing structured document images, especially in extracting structured information with complex layouts and diverse formats, making it difficult to achieve accurate semantic understanding and structured document generation.
Reinforcement learning is used to post-train a large multimodal model. By establishing a tabular knowledge database, data augmentation, and reinforcement learning-proximal policy optimization training with verifiable rewards, the OCR text recognition and structured text generation capabilities of the trained model are supplemented.
It significantly improves the accuracy of structured information extraction and semantic alignment capabilities of OCR systems under complex conditions, and achieves efficient and accurate conversion from image input to structured semantic output.
Smart Images
Figure CN121600518A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of optical character recognition technology, specifically to an optical character recognition method and system based on a multimodal large model. Background Technology
[0002] With the deepening of digital transformation, automated document processing plays an increasingly important role in many fields such as finance, taxation, logistics, and healthcare. Optical character recognition (OCR) technology, as a core means of converting paper or image documents into editable and analyzable data, has been widely applied in business scenarios such as invoice recognition, receipt entry, and contract parsing. Traditional OCR systems mostly rely on traditional supervised learning methods, achieving character recognition through the collaborative work of multiple modules. However, when processing structured document images (such as tabular images like receipts and invoices), traditional methods are insufficient in terms of layout analysis, semantic understanding, cross-regional content association, and logical reasoning, and struggle to accurately extract structured information with complex layouts and diverse formats.
[0003] The application of emerging multimodal large models has brought new breakthroughs to this technology. It has achieved excellent results in general text recognition, information extraction and other tasks. However, in terms of structured text generation, the existing model is not yet capable of generating structured documents that are consistent with the input based on the recognized text.
[0004] Therefore, there is an urgent need for a method that can specifically enhance the semantic understanding and structural extraction capabilities of multimodal large-scale OCR, so as to further improve the system's accuracy in extracting structured information and semantic alignment in complex real-world scenarios. Summary of the Invention
[0005] The technical objective of this invention is to address the above-mentioned shortcomings by providing an optical character recognition method and system based on a multimodal large model. This method and system can significantly improve the recognition accuracy of OCR systems under complex conditions of ultra-long structured text data. At the same time, it achieves efficient alignment between model output and actual results, enabling efficient, accurate, and iteratively optimizable document understanding from image input to structured semantic output.
[0006] The technical solution adopted by this invention to solve its technical problem is:
[0007] An optical character recognition method based on a multimodal large model employs reinforcement learning for post-training of the model and supplements training to address insufficient SFT training. The implementation of this method includes the following steps:
[0008] (1) Establish a table knowledge database: collect table data in image format, and construct corresponding structured data representations as labels based on the table content; the method of constructing structured data is based on the generation of multimodal large models, and manual proofreading and correction of the generated errors.
[0009] (2) Perform augmentation operations on the images (input data) in the dataset. After the augmentation is completed, divide the dataset into training set, validation set and test set and train the model.
[0010] (3) Supplement training is performed on large models that have not undergone the tasks of extracting text content from tables and pre-training structured text, including:
[0011] For large multimodal models that cannot extract text content from tables, supervised fine-tuning training is performed using the content of training set a% to enable the model to output text content from tables normally.
[0012] For large multimodal models that cannot output structured text, supervised fine-tuning training can be performed using b% of the training set to enable the model to output structured text normally (the structure does not need to be correct).
[0013] (4) Reinforcement learning-proximal policy optimization training based on the generated dataset for verifiable rewards.
[0014] After preprocessing and fine-tuning, the model is trained using a method that combines reinforcement learning based on verifiable rewards and human feedback mechanisms, achieving an efficient, accurate, and iteratively optimized document understanding system that outputs structured semantics from image input.
[0015] Furthermore, in step (1), the forms of tables mainly include invoices, bank statements, expense reimbursement forms, payrolls, budget sheets, balance sheets / profit and loss statements / cash flow statements;
[0016] Based on the concept of data augmentation, the corresponding structured data formats include formatted text formats such as Markdown, JSON, XML, and HTML, which are then used for differentiated training using different prompt words.
[0017] Furthermore, step (2) includes the following enhancement operations: flip transformation, random trimming, color dithering, translation transformation, scale transformation, contrast transformation, noise perturbation, rotation transformation, and reflection transformation.
[0018] Furthermore, training based on verifiable reinforcement learning includes:
[0019] (2.1) First, ensure that the multimodal large model has basic OCR text recognition capabilities and basic understanding of structured text semantics. Reinforcement learning is responsible for combining the two capabilities.
[0020] (2.2) The overall training of the model is based on the formula
[0021]
[0022] Different degrees of optimization, where r is the reward obtained by using y in environment x; [r φ In the specific implementation of this method, (x,y) is rewritten into three variations: truncation, constraint, and advantage to limit the training range of the model; and in practice, a training example is trained multiple times using time step t.
[0023] This formula represents the original objective of reinforcement learning, meaning that by changing the policy π... θ To maximize the value of the overall formula, β is a fixed value, x is the current environment parameter, and y is the action parameter. D KL For KL divergence function operations, π θ (y|x) is the strategy model π θ The conditional probability of generating action y given input x.
[0024] Furthermore, the supervised fine-tuning training using the content of training set a% enables the model to output the text content in the table normally, where a∈[10,25], based on the number of training sets;
[0025] The supervised fine-tuning training using the content of training set b% enables the model to output structured text normally, where b∈[10,25].
[0026] Furthermore, the reinforcement learning-proximal policy optimization training for verifiable rewards includes:
[0027] (4.1) Train the critic model using c% of the remaining structured text, where c∈[30,40]; train the critic model to perceive good and bad by augmenting the corresponding output (basically true) as rejection data;
[0028] (4.2) Formal training using reinforcement learning strategies:
[0029] S1. Model training is based on the formula:
[0030]
[0031] This formula is the original near-end learning optimization strategy formula, where: π() is the probability function of using action a in environment S under a certain strategy;
[0032] clip is a phase function, and the maximum and minimum values of the first one do not exceed (1-ε, 1+ε).
[0033] S2. Based on reinforcement learning strategy, using proximal policy optimization training as the training method, assuming the model output sequence is S T For each step t∈(1,T), the corresponding label sequence serves as the binary reward signal. The correctness of the binary reward depends on whether the character corresponding to the currently output token matches the current label;
[0034] S3. Through dynamic programming, based on the formula Adv t =(R t +γ * V t+1 -V t )+γ * λ*Adv t+1 The advantage value is calculated using a formula that is a recursive form of GAE (Generalized Advantage Estimation), where V... t R represents the output of the critic model at time step t (the critic model is responsible for measuring the output up to the current step). t Let the binary reward signal at time step t be calculated from the last output token backwards; let the advantage of the last token be Adv. T =(R T -V T );
[0035] S4. By using relative entropy, The degree of deviation between the formal constraint model and the old model, in order to The formal constraints and the degree of deviation from the comparison model.
[0036] Furthermore, the data enhancement of the corresponding output (basically real) includes operations such as adding, deleting, modifying, and reordering.
[0037] This invention also claims a multimodal large-model optical character recognition system, comprising:
[0038] The multimodal data acquisition and annotation enhancement module constructs structured data based on the generation of large multimodal models;
[0039] The model preprocessing module is used to augment the input data, divide the dataset into training, validation, and test sets, and train the model; it also provides supplementary training for large models that have not undergone the tasks of extracting text content from tables and pre-training structured text.
[0040] The reward mechanism construction module performs reinforcement learning-proximal policy optimization training for verifiable rewards based on the generated dataset;
[0041] The reinforcement learning strategy training module uses reinforcement learning strategies to train the model.
[0042] The system specifically achieves optical character recognition through the methods described above.
[0043] The present invention also claims a multimodal large-scale optical character recognition device, comprising: at least one memory and at least one processor;
[0044] The at least one memory is used to store a machine-readable program;
[0045] The at least one processor is used to call the machine-readable program to implement the above method.
[0046] The present invention also claims a computer-readable medium storing computer instructions that, when executed by a processor, implement the above-described method.
[0047] Compared with existing technologies, the optical character recognition method and system based on a multimodal large model of the present invention have the following advantages:
[0048] This invention fills the gap in the field of optical character recognition of formatted text using multimodal large models, which lacks the ability to further enhance performance through reinforcement learning. Compared to traditional methods based solely on supervised fine-tuning, this training method enables multi-step fine-tuning, rather than simply training for next-word prediction, which is more conducive to improving model accuracy and promoting model convergence. It also enhances the interpretability of the model.
[0049] This invention achieves computational advantages on both intuitive and verifiable levels. Intuition ensures the correct direction of the model in generating structured data, while verifiableness ensures the model's guarantee in generating details, thereby improving the accuracy of the generated model. Attached Figure Description
[0050] Figure 1 This is a flowchart illustrating the pre-training process of reinforcement learning-proximal policy optimization training for verifiable rewards provided in an embodiment of the present invention.
[0051] Figure 2 This is a schematic diagram of the binary determination and advantage calculation of Reward provided in an embodiment of the present invention;
[0052] Figure 3 This is a flowchart illustrating the reinforcement learning-proximal policy optimization training process provided in an embodiment of the present invention. Detailed Implementation
[0053] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.
[0054] This invention also provides an optical character recognition method based on a multimodal large model, which uses reinforcement learning to post-train the model and supplements the training for cases where SFT training is insufficient.
[0055] Currently, multimodal large-scale models are typically trained through three main stages: first, pre-training on a large-scale unlabeled corpus to learn general language capabilities; then, supervised fine-tuning (SFT) on a high-quality instruction-response pair dataset to align the model's output format and basic capabilities; finally, post-training using methods such as preference optimization (e.g., DPO) or reinforcement learning (e.g., proximal learning optimization strategies) to further align with human preferences or improve specific capabilities. This method primarily focuses on the design of post-training and includes supplementary training for situations where SFT training is insufficient, making it a comprehensive approach to improving OCR capabilities.
[0056] The implementation of this method includes the following steps:
[0057] (1) Establish a tabular knowledge database: Collect tabular data in image format, mainly including invoices, bank statements, expense reports, payrolls, budget sheets, balance sheets / profit and loss statements / cash flow statements; and construct corresponding structured data representations as labels based on the table content. Based on the data augmentation concept, the corresponding structured data formats include formatted text formats such as Markdown, JSON, XML, and HTML, which will be differentiated through different prompts. The method of constructing structured data is based on the generation of a multimodal large model, and the generated errors are manually proofread and corrected.
[0058] (2) Enhancement operations are performed on the images (input data) in the dataset, including: flip transformation, random cropping, color dithering, translation transformation, scaling transformation, contrast transformation, noise perturbation, rotation transformation, and reflection transformation. After enhancement, the dataset is divided into training set, validation set, and test set for model training.
[0059] (3) Supplement training is performed on large models that have not undergone the tasks of extracting text content from tables and pre-training structured text, including:
[0060] For large multimodal models that cannot extract text content from tables, supervised fine-tuning training is performed using the content of training set a% to enable the model to output text content from tables normally; where a∈[10,25], is set based on the number of training sets.
[0061] For large multimodal models that cannot output structured text, supervised fine-tuning training is performed using the content of training set b% to enable the model to output structured text normally (the structure does not have to be correct); where b∈[10,25].
[0062] (4) Reinforcement learning-proximal policy optimization training based on the generated dataset for verifiable rewards.
[0063] The multimodal large model is reinforced for training on tabular data, that is, the specific implementation of training based on verifiable reinforcement learning is as follows:
[0064] First, ensure that the multimodal large model has basic OCR text recognition capabilities and basic understanding of structured text semantics; reinforcement learning is responsible for combining these two capabilities.
[0065] The overall training of the model is based on the formula
[0066]
[0067] Different degrees of optimization, where r is the reward obtained by using y in environment x; [r φ In the specific implementation of this method, (x,y) is rewritten into three variations: truncation, constraint, and advantage to limit the training range of the model; and in practice, a training example is trained multiple times using time step t.
[0068] This formula represents the original objective of reinforcement learning, meaning that by changing the policy π... θ To maximize the value of the overall formula, β is a fixed value, x is the current environment parameter, and y is the action parameter. D KL For KL divergence function operations, π θ (y|x) is the strategy model π θ The conditional probability of generating action y given input x.
[0069] The reinforcement learning-proximal policy optimization training form for verifiable rewards is as follows:
[0070] The critic model is trained using c% of the remaining structured text (where c∈[30,40]) and then trained to perceive good and bad by data augmentation of the corresponding outputs (nearly true), such as adding, deleting, modifying, and reordering operations, as rejection data.
[0071] Formal training using reinforcement learning strategies:
[0072] S1. Model training is based on the formula:
[0073]
[0074] This formula is the original near-end learning optimization strategy formula, where: π() is the probability function of using action a in environment S under a certain strategy;
[0075] clip is a phase function, and the maximum and minimum values of the first one do not exceed (1-ε, 1+ε).
[0076] S2. Based on reinforcement learning strategy, using proximal policy optimization training as the training method, assuming the model output sequence is S T For each step t∈(1,T), the corresponding label sequence serves as the binary reward signal. The correctness of the binary reward depends on whether the character corresponding to the currently output token matches the current label;
[0077] S3. Through dynamic programming, based on the formula Adv t =(R t +γ * V t+1 -V t )+γ * λ*Adv t+1 The advantage value is calculated using a formula that is a recursive form of GAE (Generalized Advantage Estimation), where V... t R represents the output of the critic model at time step t (the critic model is responsible for measuring the output up to the current step). t Let the binary reward signal at time step t be calculated from the last output token backwards; let the advantage of the last token be Adv. T =(R T -V T );
[0078] S4. By using relative entropy, The degree of deviation between the formal constraint model and the old model, in order to The formal constraints and the degree of deviation from the comparison model.
[0079] like Figure 2 As shown, for the input environment S at the current time t... T Obtain the output action a. If the action matches the expected label L_t, then use the binary reward signal as an additional advantage quantity to add to the advantage calculation.
[0080] This method combines a large-scale pre-trained language model with a visual recognition model, utilizing reinforcement learning techniques to optimize the OCR system's ability to extract structured text and semantics from unstructured input. The method is particularly suitable for structured document images, such as tabular image data like receipts and invoices.
[0081] This invention also provides an optical character recognition system based on a multimodal large model, comprising:
[0082] The multimodal data acquisition and annotation enhancement module constructs structured data based on the generation of large multimodal models;
[0083] The model preprocessing module is used to augment the input data, divide the dataset into training, validation, and test sets, and train the model; it also provides supplementary training for large models that have not undergone the tasks of extracting text content from tables and pre-training structured text.
[0084] The reward mechanism construction module performs reinforcement learning-proximal policy optimization training for verifiable rewards based on the generated dataset;
[0085] The reinforcement learning strategy training module uses reinforcement learning strategies to train the model.
[0086] By introducing Human Feedback (RLHF) and Verifiable Reward (RLVR), the system can continuously learn and optimize its performance in semantic extraction, layout analysis, layout content analysis, and reasoning that combines layout semantics.
[0087] Specifically, the system achieves optical character recognition through the multimodal large model-based optical character recognition method described in the above embodiments.
[0088] (1) Establish a tabular knowledge database: Collect tabular data in image format, mainly including invoices, bank statements, expense reports, payrolls, budget sheets, balance sheets / profit and loss statements / cash flow statements; and construct corresponding structured data representations as labels based on the table content. Based on the data augmentation concept, the corresponding structured data formats include formatted text formats such as Markdown, JSON, XML, and HTML, which will be differentiated through different prompts. The method of constructing structured data is based on the generation of a multimodal large model, and the generated errors are manually proofread and corrected.
[0089] (2) Enhancement operations are performed on the images (input data) in the dataset, including: flip transformation, random cropping, color dithering, translation transformation, scaling transformation, contrast transformation, noise perturbation, rotation transformation, and reflection transformation. After enhancement, the dataset is divided into training set, validation set, and test set for model training.
[0090] (3) Supplement training is performed on large models that have not undergone the tasks of extracting text content from tables and pre-training structured text, including:
[0091] For large multimodal models that cannot extract text content from tables, supervised fine-tuning training is performed using the content of training set a% to enable the model to output text content from tables normally; where a∈[10,25], is set based on the number of training sets.
[0092] For large multimodal models that cannot output structured text, supervised fine-tuning training is performed using the content of training set b% to enable the model to output structured text normally (the structure does not have to be correct); where b∈[10,25].
[0093] (4) Reinforcement learning-proximal policy optimization training based on the generated dataset for verifiable rewards.
[0094] The multimodal large model is reinforced for training on tabular data, that is, the specific implementation of training based on verifiable reinforcement learning is as follows:
[0095] First, ensure that the multimodal large model has basic OCR text recognition capabilities and basic understanding of structured text semantics; reinforcement learning is responsible for combining these two capabilities.
[0096] The overall training of the model is based on the formula
[0097]
[0098] Different degrees of optimization, where r is the reward obtained by using y in environment x; [r φ In the specific implementation of this method, (x,y) is rewritten into three variations: truncation, constraint, and advantage to limit the training range of the model; and in practice, a training example is trained multiple times using time step t.
[0099] The reinforcement learning-proximal policy optimization training form for verifiable rewards is as follows:
[0100] The critic model is trained using c% of the remaining structured text (where c∈[30,40]) and then trained to perceive good and bad by data augmentation of the corresponding outputs (nearly true), such as adding, deleting, modifying, and reordering operations, as rejection data.
[0101] Formal training using reinforcement learning strategies:
[0102] S1. Model training is based on the formula:
[0103]
[0104] S2. Based on reinforcement learning strategy, using proximal policy optimization training as the training method, assuming the model output sequence is S T For each step t∈(1,T), the corresponding label sequence serves as the binary reward signal. The correctness of the binary reward depends on whether the character corresponding to the currently output token matches the current label;
[0105] S3. Through dynamic programming, based on the formula Adv t =(R t +γ * V t+1 -V t )+γ * λ*Adv t+1 Calculate the dominance value, V t R represents the output of the critic model at time step t (the critic model is responsible for measuring the output up to the current step). t Let the binary reward signal at time step t be calculated from the last output token backwards; let the advantage of the last token be Adv. T =(R T -V T );
[0106] S4. By using relative entropy, The degree of deviation between the formal constraint model and the old model, in order to The formal constraints and the degree of deviation from the comparison model.
[0107] This system also provides corresponding data preprocessing and model preprocessing methods, including image preprocessing and critic model training. Compared to traditional supervised learning, it significantly improves the recognition accuracy of OCR systems under complex conditions of ultra-long structured text data. Furthermore, through human feedback mechanisms and verifiable rewards, it objectively achieves efficient alignment between model output and actual results, demonstrating good scalability and practical application value.
[0108] This invention also provides an optical character recognition device based on a multimodal large model, comprising: at least one memory and at least one processor;
[0109] The at least one memory is used to store a machine-readable program;
[0110] The at least one processor is used to call the machine-readable program to implement the optical character recognition method based on a multimodal large model as described in the above embodiments.
[0111] This invention also provides a computer-readable medium storing computer instructions. When executed by a processor, these computer instructions implement the optical character recognition method based on a multimodal large model described in the above embodiments. Specifically, a system or apparatus equipped with a storage medium storing software program code that implements the functions of any of the embodiments described above, and enabling the computer (or CPU, NPU, or GPU) of the system or apparatus to read and execute the program code stored in the storage medium.
[0112] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.
[0113] Examples of storage media used to provide program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.
[0114] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.
[0115] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.
[0116] The present invention has been shown and described in detail above with reference to the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above embodiments, those skilled in the art will know that more embodiments of the present invention can be obtained by combining the code review methods in the different embodiments. These embodiments are also within the protection scope of the present invention.
Claims
1. A method for optical character recognition based on a multimodal large model, characterized in that, The model is post-trained using reinforcement learning, and supplementary training is performed to address insufficient SFT training. The implementation of this method includes the following steps: (1) Establish a table knowledge database: collect table data in image format, and construct corresponding structured data representations as labels based on the table content; the method of constructing structured data is based on the generation of multimodal large models, and manual proofreading and correction of the generated errors. (2) Perform enhancement operations on the images in the dataset. After enhancement, divide the dataset into training set, validation set and test set and train the model. (3) Supplement training is performed on large models that have not undergone the tasks of extracting text content from tables and pre-training structured text, including: For large multimodal models that cannot extract text content from tables, supervised fine-tuning training is performed using the content of training set a% to enable the model to output text content from tables normally. For large multimodal models that cannot output structured text, supervised fine-tuning training can be performed using b% of the training set to enable the model to output structured text normally. (4) Reinforcement learning-proximal policy optimization training based on the generated dataset for verifiable rewards.
2. The optical character recognition method based on a multimodal large model according to claim 1, characterized in that, The form of the table in step (1) includes invoices, bank statements, expense reimbursement forms, payrolls, budget sheets, balance sheets / profit and loss statements / cash flow statements; Based on the concept of data augmentation, the corresponding structured data formats include formatted text formats such as Markdown, JSON, XML, and HTML, which are then used for differentiated training using different prompt words.
3. The optical character recognition method based on a multimodal large model according to claim 1, characterized in that, The enhancement operations in step (2) include: flip transformation, random trimming, color dithering, translation transformation, scale transformation, contrast transformation, noise perturbation, rotation transformation, and reflection transformation.
4. The optical character recognition method based on a multimodal large model according to claim 1, characterized in that, Training based on verifiable reinforcement learning includes: (2.1) First, ensure that the multimodal large model has basic OCR text recognition capabilities and basic understanding of structured text semantics. Reinforcement learning is responsible for combining the two capabilities. (2.2) The overall training of the model is based on the formula Different degrees of optimization, where r is the reward obtained by using y in environment x; [r φ In the specific implementation of this method, (x,y) is rewritten into three variations: truncation, constraint, and advantage to limit the training range of the model; and in practice, a training example is trained multiple times using time step t.
5. The optical character recognition method based on a multimodal large model according to claim 1, characterized in that, The supervised fine-tuning training using the content of training set a% enables the model to output the text content in the table correctly, where a∈[10,25], based on the number of training sets; The supervised fine-tuning training using the content of training set b% enables the model to output structured text normally, where b∈[10,25].
6. The optical character recognition method based on a multimodal large model according to claim 1, characterized in that, The reinforcement learning-proximal policy optimization training for verifiable rewards includes: (4.1) Train the critic model using c% of the remaining structured text, where c∈[30,40]; train the critic model to perceive good and bad by augmenting the corresponding outputs as rejection data; (4.2) Formal training using reinforcement learning strategies: S1. Model training is based on the formula: S2. Based on reinforcement learning strategy, using proximal policy optimization training as the training method, assuming the model output sequence is S T For each step t∈(1,T), the corresponding label sequence serves as the binary reward signal. The correctness of the binary reward depends on whether the character corresponding to the currently output token matches the current label; S3. Through dynamic programming, based on the formula Adv t =(R t +γ*V t+1 -V t )+γ*λ*Adv t+1 Calculate the dominance value, V t R represents the output of the critic model at time step t (the critic model is responsible for measuring the output up to the current step). t Let the binary reward signal at time step t be calculated from the last output token backwards; let the advantage of the last token be Adv. T =(R T -V T ); S4. By using relative entropy, The degree of deviation between the formal constraint model and the old model, in order to The formal constraints and the degree of deviation from the comparison model.
7. The optical character recognition method based on a multimodal large model according to claim 6, characterized in that, The data enhancements to the corresponding output include operations such as adding, deleting, modifying, and reordering.
8. An optical character recognition system based on a multimodal large model, characterized in that, include: The multimodal data acquisition and annotation enhancement module constructs structured data based on the generation of large multimodal models; The model preprocessing module is used to augment the input data, divide the dataset into training, validation, and test sets, and train the model; it also provides supplementary training for large models that have not undergone the tasks of extracting text content from tables and pre-training structured text. The reward mechanism construction module performs reinforcement learning-proximal policy optimization training for verifiable rewards based on the generated dataset; The reinforcement learning strategy training module uses reinforcement learning strategies to train the model. The system specifically achieves optical character recognition through the method described in any one of claims 1 to 7.
9. An optical character recognition device based on a multimodal large model, characterized in that, include: At least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to invoke the machine-readable program to implement the method according to any one of claims 1 to 7.
10. A computer-readable medium, characterized in that, The computer-readable medium stores computer instructions that, when executed by a processor, implement the method described in any one of claims 1 to 7.