Optical character recognition model reinforcement learning optimization method and device

Abstract

The application discloses an optical character recognition model reinforcement learning optimization method and device, and relates to the technical field of optical character recognition, wherein the method comprises the following steps: obtaining a benchmark OCR model and a pre-training model with the function of optical character recognition; introducing factual rewards, performing reinforcement learning optimization on the pre-training model, and obtaining an optimized model; the factual rewards comprise semantic consistency rewards, logical fact verification rewards, and output length penalty terms; wherein the semantic consistency rewards reflect the consistency degree of the pre-training model output and the labeled true value; the logical fact verification rewards comprise rewards for the semantic logic self-consistency degree of the pre-training model output and verification of the structured fields in the pre-training model output by using a deterministic rule engine; and the output length penalty terms reflect the deviation of the length of the pre-training model output from a reference length; and the reference length is output by the benchmark OCR model. The application can improve the accuracy of image text recognition and the robustness of the model.

Description

Technical Field

[0001] This invention relates to the fields of optical character recognition, multimodal large models, and deep learning technologies, and particularly to a reinforcement learning optimization method and apparatus for optical character recognition models. Background Technology

[0002] In traditional prosecutorial work, prosecutors manually review paper or electronic case files page by page, which is time-consuming and labor-intensive. With the development of artificial intelligence technology, optical character recognition (OCR) technology can quickly scan and recognize case materials (such as paper documents and electronic data), automatically extract key information (such as information about criminal suspects and evidence lists), and generate structured summaries, significantly shortening the review time. However, existing OCR technology still has significant performance bottlenecks in complex document scenarios (such as invoices, forms, contracts, academic papers, and scanned documents). Currently, the mainstream OCR algorithms are mainly divided into two categories: one is the traditional small OCR model based on a visual detection model plus a text classification model, such as Tesseract (an open-source optical character recognition engine); the other is the large multimodal model based on the Transformer architecture that has emerged in recent years. Although the traditional small OCR model has the advantages of speed and lightweight, it has poor adaptability to complex scenarios; while the large multimodal model has powerful semantic understanding capabilities, it is prone to outputting text that is "semantically reasonable but factually incorrect" (such as mismatched amounts, times, and units).

[0003] Therefore, a new model algorithm is needed to address the shortcomings of the existing technology. Summary of the Invention

[0004] This invention provides a reinforcement learning optimization method for an optical character recognition model to improve the accuracy of image text recognition and enhance the robustness of the model. The method includes:

[0005] Obtain a baseline OCR model and a pre-trained model with optical character recognition capabilities; the baseline OCR model is used as a benchmark to verify the output of the pre-trained model.

[0006] A factual reward is introduced to optimize the pre-trained model using reinforcement learning, resulting in an optimized model. This factual reward includes a semantic consistency reward, a logical fact verification reward, and a production length penalty term.

[0007] Semantic consistency reward reflects the degree of consistency between the output of the pre-trained model and the labeled truth.

[0008] The logical fact verification reward includes a reward for the semantic logical consistency of the pre-trained model output and a reward for verifying the structured fields in the pre-trained model output using a deterministic rule engine; the deterministic rule engine is used to determine whether the input fields meet the preset specifications.

[0009] The output length penalty term reflects the deviation between the length output by the pre-trained model and the reference length, which is output by the baseline OCR model.

[0010] This invention also provides a reinforcement learning optimization device for an optical character recognition model to improve the accuracy of image text recognition and enhance the robustness of the model. The device includes:

[0011] A pre-training module is used to obtain a baseline OCR model and a pre-trained model with optical character recognition capabilities; the baseline OCR model is used as a benchmark to verify the output of the pre-trained model.

[0012] The reinforcement learning optimization module is used to introduce fact rewards and perform reinforcement learning optimization on the pre-trained model to obtain an optimized model. The fact rewards include semantic consistency rewards, logical fact verification rewards, and output length penalties. Among them, the semantic consistency reward reflects the degree of consistency between the output of the pre-trained model and the labeled truth value; the logical fact verification reward includes rewards for the semantic and logical self-consistency of the output of the pre-trained model and verification of the structured fields in the output of the pre-trained model using a deterministic rule engine; the deterministic rule engine is used to determine whether the input fields meet the preset specifications; the output length penalty reflects the deviation between the length of the output of the pre-trained model and the reference length; the reference length is output by the benchmark OCR model.

[0013] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described reinforcement learning optimization method for optical character recognition models.

[0014] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described reinforcement learning optimization method for optical character recognition models.

[0015] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described reinforcement learning optimization method for optical character recognition models.

[0016] In this embodiment of the invention, a fact reward is introduced into the model reinforcement learning. This fact reward integrates three dimensions: semantic consistency reward, logical fact verification reward, and output length penalty. Semantic consistency and logical fact verification rewards improve fact consistency, increasing the matching degree between the model's output semantics and image content by 8.8 percentage points. Generation control is enhanced, and the text length penalty effectively avoids redundancy or missing characters, demonstrating strong robustness. Furthermore, it is adaptable to various document types such as invoices, contracts, archives, and forms, exhibiting strong scalability. Through the method of this embodiment, the optimized model can maintain high accuracy and high consistency in multi-scenario document recognition tasks, achieving intelligent understanding "from image to real semantic text," providing highly reliable technical support for intelligent case file processing, clue and evidence identification. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0018] Figure 1 This is a flowchart illustrating the reinforcement learning optimization method for the optical character recognition model in an embodiment of the present invention.

[0019] Figure 2 This is a flowchart of an example of the reinforcement learning optimization method for optical character recognition models in this invention.

[0020] Figure 3 This is a flowchart illustrating another example of the reinforcement learning optimization method for the optical character recognition model in this invention.

[0021] Figure 4 This is a schematic diagram of the reinforcement learning optimization device for the optical character recognition model in an embodiment of the present invention;

[0022] Figure 5 This is a schematic diagram of a computer device in an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0024] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations.

[0025] First, the technical terms involved in the embodiments of the present invention will be explained.

[0026] Optical Character Recognition (OCR): OCR is a technology that converts printed, handwritten, or digital text into a computer-editable and processable text form from digital images (such as photos, screenshots, etc.).

[0027] Document Layout Analysis (NBR) is a technique that analyzes and extracts the content structure of a document image or electronic document. It aims to identify different areas within the document (such as headings, paragraphs, tables, images, headers, footers, etc.) and determine their spatial layout and logical relationships on the page. This technique helps in understanding the overall structure of the document and provides a foundation for subsequent document processing (such as information extraction, content summarization, format conversion, etc.).

[0028] Group Relative Policy Optimization (GRPO) is a reinforcement learning-based policy optimization algorithm designed to improve the performance of large language models in complex tasks such as mathematical reasoning and programming. Its core idea is to optimize the policy through relative comparisons of samples within a group, rather than relying on absolute reward values, thereby reducing computational costs and improving training stability.

[0029] Proximal Policy Optimization (PPO): PPO is a reinforcement learning algorithm that prevents training from collapsing by limiting the difference between the new and old policies at each policy update (i.e., "proximal constraint").

[0030] Currently, mainstream OCR algorithms are mainly divided into two categories: one is the traditional small OCR model based on visual detection model plus text classification model; the other is the large multimodal model based on Transformer architecture that has emerged in recent years.

[0031] Traditional small OCR models typically employ a two-stage or multi-stage pipeline architecture of "text region detection and character or text line recognition." Their advantages are mainly reflected in the following aspects: 1. Low resource consumption. 2. High potential for accuracy optimization in specific scenarios. 3. Fast inference speed. However, small models also have significant disadvantages: 1. Poor robustness of multi-module collaboration; the detection and recognition modules are independent of each other, and errors in the preceding detection module (such as missed text regions or positioning offsets) will directly propagate to the recognition module, leading to a decrease in overall accuracy. For example, for text with excessively large tilt angles (>45°) or occlusions, the detection module is prone to positioning deviations, resulting in complete failure of the recognition results. 2. Insufficient generalization in complex scenarios: They have weak processing capabilities for non-standard documents (such as handwriting, artistic fonts, and blurred text), complex layouts (such as multilingual mixed typesetting and academic papers with dense text and images), or adverse imaging conditions (such as low light, reflection, and distortion), resulting in a significant decrease in recognition accuracy. Furthermore, they require separate development of adaptation modules for different scenarios, leading to poor generalization. 3. Lack of semantic understanding: Traditional OCR small models can only complete the mechanical conversion from "image to text" and cannot combine the semantic logic of the document to correct recognition errors (such as when "amount: 1000 yuan" is misidentified as "amount: 1000 none", it cannot be corrected through semantic rationality judgment), and it is also difficult to associate context to understand the logical relationship of cross-regional text (such as the correspondence between "table header - content" in a table).

[0032] The emergence of large-scale multimodal OCR models based on Transformer decoders has effectively solved the problem of small OCR models. It adopts an "end-to-end" architecture, processing image information through a unified encoder (such as Vision Transformer, a computer vision model based on the Transformer architecture), and combining it with a pre-trained language model to achieve a direct mapping from "image input to text output." The model also integrates document layout analysis capabilities, simultaneously outputting text content and region semantic types (such as titles, tables, and paragraphs). However, while solving the problems of small models, large-scale multimodal models bring new challenges: 1. Extremely high resource consumption and high deployment threshold: Large-scale multimodal models typically have billions to hundreds of billions of parameters, requiring high-performance GPUs (such as A100 and H100) for inference, consuming tens of gigabytes of memory, making them unsuitable for edge devices, and consuming high power, thus unsuitable for low-cost, lightweight scenarios. 2. Generative large models suffer from the same problem as large models in OCR tasks: the model illusion problem. In multimodal large models, illusion refers to the model outputting text content that is unrelated to or contradicts the input image. Specifically, it manifests in three ways: First, generative errors: fabricating text that does not exist in the image (such as filling in a blurred area with non-existent content); second, semantic deviation: misinterpreting the layout structure (such as identifying a table title as a paragraph); and third, overgeneralization: making incorrect associations based on local features (such as identifying an official seal image as text content).

[0033] Existing optical character recognition technologies still have significant performance bottlenecks in complex document scenarios (such as invoices, forms, contracts, academic papers, scanned documents, etc.).

[0034] In addition, while current reinforcement learning training methods (such as PPO, GRPO, etc.) can improve the quality of language model generation, they still have some problems when directly applied to OCR tasks. For example, the reward signal is singular and one-sided, which cannot ensure the consistency between the recognized text and the real image content. They also lack objective comparison objects and make it difficult to stably evaluate the direction of improvement during training.

[0035] To address the aforementioned challenges, this invention proposes a reinforcement learning optimization method for optical character recognition (OCR) models based on fact-based reward enhancement. This method innovatively combines a group-relative policy optimization algorithm, a text length penalty mechanism, and a baseline comparison with traditional OCR models to construct a quantifiable reward system for factual consistency. This system iteratively optimizes the model using reinforcement learning, based on the degree of alignment between the factual reward output and the image semantics, enabling the OCR model not only to accurately recognize text but also to understand and maintain semantic logical consistency.

[0036] Figure 1 This is a flowchart illustrating the reinforcement learning optimization method for the optical character recognition model in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0037] Step 101: Obtain a baseline OCR model and a pre-trained model with optical character recognition capabilities; the baseline OCR model is used as a benchmark to verify the output of the pre-trained model.

[0038] Step 102: Introduce fact rewards and perform reinforcement learning optimization on the pre-trained model to obtain the optimized model. The fact rewards include semantic consistency rewards, logical fact verification rewards, and output length penalties. The semantic consistency reward reflects the consistency between the output of the pre-trained model and the labeled truth value. The logical fact verification reward includes rewards for the semantic and logical self-consistency of the output of the pre-trained model and verification of the structured fields in the output of the pre-trained model using a deterministic rule engine. The deterministic rule engine is used to determine whether the input fields meet the preset specifications. The output length penalty reflects the deviation between the length of the output of the pre-trained model and the reference length. The reference length is output by the benchmark OCR model.

[0039] The following describes in detail the reinforcement learning optimization method for the optical character recognition model in the embodiments of the present invention.

[0040] The method proposed in this invention aims to solve the problems of image-text fact deviation and training process oscillation in large OCR models under complex text scenarios. By integrating group relative policy optimization algorithm, text length penalty mechanism and traditional OCR model baseline comparison, a reinforcement learning framework that can be optimized end-to-end is formed.

[0041] First, prepare the basic large model and the "anchor point guidance" baseline module.

[0042] The basic large model is a pre-trained model that possesses optical character recognition capabilities. In one embodiment, obtaining a pre-trained model with optical character recognition capabilities may include:

[0043] A pre-trained model is obtained by training a visual language model using labeled image and text pairs. The visual language model includes a visual encoder and a language decoder. The visual encoder is used to encode the input image into visual features, and the language decoder is used to generate text based on the visual features.

[0044] During implementation, a large-scale image-text pair dataset is selected, and features are extracted using a visual encoder and combined with a language encoder to complete supervised training, thereby constructing a multimodal language model M0 with basic OCR mapping capabilities, i.e., a pre-trained model.

[0045] The baseline module introduces an existing traditional OCR model as a benchmark OCR model, providing a reference output for the input image. Specifically, the benchmark OCR model can be used as a benchmark to validate the output of the pre-trained model.

[0046] This baseline OCR model provides a reference baseline for subsequent model training. During the reinforcement learning phase, it guides structured alignment to evaluate deviations in the format and length of the generated text.

[0047] Convergence robustness guarantee: In this embodiment of the invention, the baseline provides a stable, low-noise target, which effectively mitigates random fluctuations during the policy update process and ensures that the model maintains the correctness of the layout while improving semantics.

[0048] Second, rewards based on facts.

[0049] The authenticity of the model's output is ensured by calculating factual rewards. These rewards consist of three parts.

[0050] 1. Semantic consistency reward reflects the degree of consistency between the output of the pre-trained model and the labeled truth value.

[0051] In one embodiment, the semantic consistency reward is obtained by evaluating the consistency between the pre-trained model output and the labeled ground truth through edit distance and semantic embedding similarity.

[0052] Edit distance and semantic embedding similarity can simultaneously verify both surface structure and deep meaning. Edit distance quantifies character-level differences, assessing literal overlap of text with minimal additions, deletions, and modifications. Semantic embedding maps text to high-dimensional vectors, and cosine similarity between vectors captures consistency between context and intent. Finally, the two methods can be weighted and fused to obtain a more comprehensive and robust semantic consistency reward.

[0053] 2. Logical fact verification reward adopts a dual-path verification architecture of "rule engine - discriminative model" to reduce training feedback latency and ensure logical rigor.

[0054] Specifically, a deterministic rule engine is used to validate the structured fields in the output of the pre-trained model; the deterministic rule engine is used to determine whether the input fields meet the preset specifications.

[0055] For example, for fields with standard topological structures, such as the consistency of monetary values ​​and date format specifications, the preset deterministic rule engine is called first for verification, and the reward deviation is directly output.

[0056] For unstructured semantic logic, such as the logical consistency between the description of the crime and the chain of evidence in the case file, a discriminative model method is adopted.

[0057] For example, the reward for the semantic logical consistency of the pre-trained model output is obtained as follows: the semantic logical consistency of the pre-trained model output is evaluated using a logic evaluator; the logic evaluator uses preset rule judgment or a lightweight neural network model to judge the logical consistency, and the model parameter magnitude of the logic evaluator is lower than that of the pre-trained model.

[0058] The logic evaluator is a lightweight discriminative special-purpose language model. This special-purpose model is different from the generation logic of the main model. Its parameter scale is usually 5%-10% of that of the main model, and it is frozen during training.

[0059] Furthermore, for example, the reward for the semantic logical consistency of the pre-trained model output is obtained as follows:

[0060] The output text of the pre-trained model is converted into a vector, and the vector is compared with the statement vectors in the preset business logic knowledge base by cosine similarity. Based on the comparison result, a reward is output for the semantic and logical self-consistency of the output of the pre-trained model. The statement vectors in the preset business logic knowledge base are obtained based on text with accurate logical expression.

[0061] Discriminative models can also calculate logically consistent reward scores by converting the model's output text into high-dimensional logical vectors and comparing them with a pre-defined business logic knowledge base using cosine similarity. This "separation of generation and discrimination" architecture effectively avoids the self-loop problem during training, significantly improving the accuracy of factual correctness rewards while ensuring the training throughput and optimization stability of the GRPO reinforcement learning framework.

[0062] 3. The output length penalty term reflects the deviation between the length output by the pre-trained model and the reference length; the reference length is output by the benchmark OCR model.

[0063] In this embodiment, the output length penalty term is obtained according to the following formula:

[0064] ;

[0065] In the formula, P l This is a penalty for output length. This is the penalty coefficient; L 0 This is a reference length used to suppress redundant generation and ensure concise text. L l This is the length of the output of the pre-trained model.

[0066] The output length penalty term can control the output length and prevent redundant or excessively short text.

[0067] 4. Composite optimization mechanism.

[0068] In step 102, during the reinforcement learning optimization process, a factual reward is calculated for each candidate text sequence, specifically including:

[0069] The fact reward includes semantic consistency reward and logical fact verification reward. The semantic consistency reward, logical fact verification reward and output length penalty term are fused together using the assigned weights to calculate the fact reward for each candidate text sequence.

[0070] For example, the formula for calculating the comprehensive reward is: R f = × R s + × R c + P l ;

[0071] in, R f The fact reward for the candidate text sequence represents the comprehensive reward score calculated for a single candidate sequence. It is a weighted combination of three dimensions: semantic consistency, factual correctness, and length penalty. It is used to guide the model to optimize both semantic accuracy and logical consistency during the generation process.

[0072] R s , R c , P l These are semantic consistency reward, logical fact verification reward, and output length penalty;

[0073] The weight of the semantic consistency reward is specifically the weight ratio factor allocated to the semantic consistency reward, which is used to adjust the model's emphasis on the degree of matching of labeled truth values ​​at the text level.

[0074] The weight of the logical fact verification reward is specifically the weight ratio factor allocated to the logical fact verification reward. In scenarios requiring high-precision recognition, this coefficient is usually increased to strengthen the constraint of logical verification.

[0075] , The sum equals 1, weight , Dynamically configure based on application scenarios, adding features for high-precision document processing scenarios. The proportion is used to eliminate factual bias.

[0076] The formula for calculating factual rewards ensures that the model optimizes both semantic correctness and logical consistency during the generation process, eliminating factual bias.

[0077] Third, reinforcement learning optimization based on improved GRPO.

[0078] In this embodiment of the invention, improvements were made to the traditional GRPO algorithm by introducing baseline contrast loss and length penalty terms to enhance stability and convergence speed.

[0079] Step 102 introduces factual rewards to optimize the pre-trained model through reinforcement learning, resulting in an optimized model that may include:

[0080] Based on the group relative policy optimization algorithm, a fact reward is introduced to optimize the pre-trained model through reinforcement learning, resulting in an optimized model. The pre-trained model generates multiple candidate text sequences for each input image, calculates a fact reward for each candidate text sequence, calculates the loss function based on the fact reward, and calculates the gradient and updates the model parameters through backpropagation.

[0081] During the reinforcement learning optimization phase, the pre-trained model from step 101 is invoked to form the initial policy distribution. For each input image x Using the initial policy distribution Inference sampling is performed, and k different candidate text sequences are generated in parallel. These k sequences constitute the within-group sample space of the GRPO algorithm, which is used for subsequent normalized reward calculation based on relative ranking to smoothly guide the policy gradient.

[0082] According to the comprehensive calculation formula for factual rewards, for each (i is any integer between 1 and k, Calculate the fact reward for any of the k sequences. R f ( );

[0083] The samples within the group are ranked relative to each other, and a normalized reward is obtained. ;

[0084] Update policy gradient:

[0085] ;

[0086] in, This represents the gradient operator, used to adjust the model parameters. Find the partial derivatives to indicate the direction of parameter updates;

[0087] Representing the objective function (or expected reward), the goal of reinforcement learning is to adjust the parameters... To maximize the value of the function;

[0088] The policy gradient, which is the derivative of the objective function with respect to the model parameters, is used to guide the iterative update of the parameters.

[0089] E i [...] indicates samples within the group Finding the expected value is usually represented by averaging the update values ​​of k candidate output sequences in the calculation;

[0090] This represents the normalized reward value of the i-th candidate text sequence. It is calculated by relative ranking and normalization of the factual rewards of all samples within the group, reflecting the relative merit of the output within the group.

[0091] log represents the natural logarithm function, which is used in the policy gradient algorithm to transform the probability product into a summation form, making gradient calculation easier;

[0092] x This represents the input image.

[0093] Furthermore, based on the group relative policy optimization algorithm, factual rewards are introduced to optimize the pre-trained model through reinforcement learning, resulting in an optimized model, which may include:

[0094] At each training step, the output of the benchmark OCR model is introduced as a structured comparison baseline, and KL divergence constraints are added:

[0095] ;

[0096] in, The comparison constraint loss refers to the comparison constraint term between the output of the pre-trained model and the reference baseline OCR model. This represents the dynamic weight coefficient, which is used to adjust the proportion of KL divergence constraint in the total loss. Its value evolves dynamically with the training progress. y l 、y t These are the outputs of the pre-trained model and the baseline OCR model, respectively. KL ( ) indicates divergence calculation.

[0097] Definitions of total loss function and constraint terms:

[0098] To prevent policy collapse during training, this embodiment of the invention introduces a comparison constraint loss in each training step. The final optimized total loss function Defined as:

[0099] .

[0100] In this embodiment of the invention, controllable generation is achieved through "fact reinforcement + structural comparison", which enables the model to generate semantically accurate results without deviating from the OCR format specifications.

[0101] Fourth, multi-stage intensive training and distillation deployment.

[0102] The embodiments of the present invention employ a progressive reinforcement training strategy.

[0103] First, the step of training the visual language model using labeled image and text data to obtain a pre-trained model may include: freezing the visual encoder and fine-tuning the language decoder when training the visual language model using labeled image and text data.

[0104] Then, reinforcement learning is used for optimization.

[0105] Finally, in the distillation stage, after obtaining the optimized model, the method may further include: distilling the optimized model as the teacher model to obtain a lightweight student model, deploying the lightweight student model on a designated terminal; and retaining factual rewards during the distillation process.

[0106] Figure 2 This is a flowchart illustrating an example of a reinforcement learning optimization method for an optical character recognition model in an embodiment of the present invention. (Refer to...) Figure 2 Progressive reinforcement training strategies include:

[0107] Phase 1: Freeze the visual encoder and fine-tune the language decoder;

[0108] Phase 2: Enable full model parameter updates and introduce baseline comparison with benchmark OCR models;

[0109] The third stage, the distillation stage, involves transferring the output of the reinforcement learning-optimized model to a lightweight student model for deployment on the edge or in a document batch processing system.

[0110] In this process, factual reward signals are preserved, allowing the student model to inherit the factual consistency constraint. The objective function for the distillation stage is:

[0111] ;

[0112] In the formula, The objective function for the distillation stage; This represents a lightweight student model. This represents the model optimized through reinforcement learning; y in both and represents the image x Enter to The output, y in both and represents the image x Enter to The output, Indicates KL divergence constraint, Representing an image x Enter to The output of the fact reward.

[0113] This stage ensures that the lightweight model can still maintain high consistency and high accuracy in text output.

[0114] Figure 3 This is a flowchart illustrating another example of the reinforcement learning optimization method for the optical character recognition model in this invention. (Refer to...) Figure 3 This method includes an improved GRPO workflow, a reward calculation component, and reinforcement learning optimization. The improved GRPO workflow includes large-scale model pre-training to obtain a multimodal language model M0, and parallel generation of k candidate texts y from a single input image. i The same image is output by the baseline OCR model. t The labeled truth value y_gt is obtained. In the reward calculation section, the semantic consistency reward, logical fact verification reward, and output length penalty are calculated separately, and then the composite reward is calculated. R f = × R s + × R c + P l In the reinforcement learning optimization, we introduce intra-group ranking, reward normalization, and KL divergence constraints, and finally update the model M0.

[0115] Example 1: Implementation of GRPO reinforcement learning based on fact-based rewards.

[0116] 1.1 Network structure design.

[0117] The base model consists of a visual encoder and a language decoder. The traditional OCR baseline uses an open-source OCR toolkit.

[0118] The core structure of the reinforcement learning phase includes semantic consistency rewards, logical fact verification rewards, and output length penalties.

[0119] 1.2. Factual rewards are achieved.

[0120] Calculate the semantic consistency reward, logical fact verification reward, and output length penalty separately.

[0121] Example of reward calculation:

[0122] If the output amount field "1200" matches the image, then Rs =1.0;

[0123] If it is identified as "12000" or "1200 yuan", then Rs =-0.5, and P l =-0.2.

[0124] 1.3 Training Strategies

[0125] Two-stage training is employed:

[0126] Phase 1: Freeze the visual encoder and optimize only the language decoder;

[0127] Phase 2: Enable full-parameter GRPO optimization and introduce KL constraints.

[0128] The optimizer is AdamW, the learning rate is 1e-5, and the batch size is 8.

[0129] Example 2: Progressive reinforcement training strategy and parameter settings.

[0130] Phase 1 (Supervised Pre-training, 20% of rounds):

[0131] Cross-entropy training is performed on labeled text-image pairs, and fact rewards are disabled.

[0132] Phase 2 (GRPO training, 60% of rounds):

[0133] Introducing fact-based rewards and text length penalties, dynamically adjusting dynamic weighting coefficients. ,in p Based on training progress. p It is calculated based on the number of epochs.

[0134] Phase 3 (Reward Distillation Fine-tuning, 20% of rounds):

[0135] by Distillation of teacher model to ,reserve information.

[0136] The total loss function is: .

[0137] In the formula, Supervised training loss, referring to standard cross-entropy loss, is mainly used for maintaining basic capabilities in the first stage of supervised pre-training or in multi-stage training. Its technical purpose is to enable the model to learn basic visual feature extraction and text generation mapping capabilities through labeled "image-text" pairs, thereby obtaining a benchmark multimodal model M0 with basic OCR recognition capabilities.

[0138] Reinforcement learning gradient loss refers to the policy gradient loss, which is usually calculated by the algorithm during the reinforcement learning phase.

[0139] This strategy ensures stable model training and avoids fact reward oscillations or overfitting.

[0140] Example 3: Experimental Results and Performance Evaluation.

[0141] Testing on a self-built invoice and contract OCR dataset:

[0142] Training data: approximately 500,000 image-text pairs, divided into 400,000 printed images and 100,000 handwritten images;

[0143] Baseline model: Q-VL (unenhanced);

[0144] Comparison metrics: character-level accuracy, edit distance, and factual consistency.

[0145] The results are shown in Table 1 below.

[0146] Table 1

[0147]

[0148] The results show that the method of the present invention significantly improves the consistency of textual facts and the controllability of generation, and the output is more concise and accurate.

[0149] The embodiments of the present invention enhance the factual consistency of OCR output by introducing group relative strategy optimization, text length penalty and traditional OCR comparison baseline.

[0150] The factual reward integrates three dimensions: semantic consistency reward, logical fact verification reward, and output length penalty, forming a composite optimization signal.

[0151] The output of a traditional OCR model is used as a structural benchmark, and KL divergence constraints are used to ensure the stability of the generated text structure during the reinforcement learning stage.

[0152] During training, a progressive reinforcement learning strategy is adopted, and the GRPO and reward weights are dynamically adjusted to ensure training stability and generalization ability.

[0153] Ultimately, a lightweight OCR model is formed through knowledge distillation, which is suitable for low-computing-power deployment environments.

[0154] The embodiments of this invention significantly improve the recognition accuracy and factual consistency of large OCR models, achieving the goal of "recognition equals understanding," and providing highly reliable technical support for intelligent case file processing, clue and evidence identification. The invention offers the following beneficial technical effects:

[0155] Improved factual consistency: The model output semantics improved its matching accuracy with image content by 8.8 percentage points;

[0156] Enhanced generation control: Text length penalty effectively avoids redundancy or missing words;

[0157] Improved training stability: GRPO combined with baseline constraints avoids reward oscillations and achieves smooth convergence;

[0158] Highly scalable: The method can be adapted to various document types such as invoices, contracts, archives, and forms;

[0159] Lightweight and easy to deploy: Distillation model parameters are reduced by 42%, and inference speed is increased by 1.8 times.

[0160] This invention also provides an optical character recognition model reinforcement learning optimization device, as described in the following embodiments. Since the principle by which this device solves the problem is similar to that of the optical character recognition model reinforcement learning optimization method, the implementation of this device can refer to the implementation of the optical character recognition model reinforcement learning optimization method; repeated details will not be elaborated further.

[0161] Figure 4 This is a schematic diagram of the reinforcement learning optimization device for the optical character recognition model in an embodiment of the present invention, as shown below. Figure 4 As shown, the device 400 includes:

[0162] The pre-training module 401 is used to obtain a baseline OCR model and a pre-trained model with optical character recognition function; the baseline OCR model is used as a benchmark to verify the output of the pre-trained model.

[0163] The reinforcement learning optimization module 402 is used to introduce fact rewards to optimize the pre-trained model through reinforcement learning, resulting in an optimized model. The fact rewards include semantic consistency rewards, logical fact verification rewards, and output length penalties. The semantic consistency reward reflects the consistency between the pre-trained model output and the labeled truth value. The logical fact verification reward includes rewards for the semantic and logical self-consistency of the pre-trained model output and for verifying the structured fields in the pre-trained model output using a deterministic rule engine. The deterministic rule engine is used to determine whether the input fields meet preset specifications. The output length penalty reflects the deviation between the length of the pre-trained model output and the reference length, which is output by the baseline OCR model.

[0164] In one embodiment, the semantic consistency reward is obtained by evaluating the consistency between the pre-trained model output and the labeled ground truth through edit distance and semantic embedding similarity.

[0165] In one embodiment, the reward for the semantic logical consistency of the pre-trained model output is obtained as follows:

[0166] A logic evaluator is used to assess the semantic and logical consistency of the output of the pre-trained model. The logic evaluator uses preset rule judgment or a lightweight neural network model to judge the logical consistency. The number of model parameters of the logic evaluator is lower than that of the pre-trained model.

[0167] In one embodiment, the reward for the semantic logical consistency of the pre-trained model output is obtained as follows:

[0168] The output text of the pre-trained model is converted into a vector, and the vector is compared with the statement vectors in the preset business logic knowledge base by cosine similarity. Based on the comparison result, a reward is output for the semantic and logical self-consistency of the output of the pre-trained model. The statement vectors in the preset business logic knowledge base are obtained based on text with accurate logical expression.

[0169] In one embodiment, the output length penalty term is obtained according to the following formula:

[0170] ;

[0171] In the formula, P l As a penalty for output length, The penalty coefficient is... L 0 For reference length, L l This is the length of the output of the pre-trained model.

[0172] In one embodiment, the reinforcement learning optimization module 402 is specifically used for:

[0173] Based on the group relative policy optimization algorithm, a fact reward is introduced to optimize the pre-trained model through reinforcement learning, resulting in an optimized model. The pre-trained model generates multiple candidate text sequences for each input image, calculates a fact reward for each candidate text sequence, calculates the loss function based on the fact reward, and calculates the gradient and updates the model parameters through backpropagation.

[0174] In one embodiment, the reinforcement learning optimization module 402 is specifically used for:

[0175] The fact reward includes semantic consistency reward and logical fact verification reward. The semantic consistency reward, logical fact verification reward and output length penalty term are fused together using the assigned weights to calculate the fact reward for each candidate text sequence.

[0176] In one embodiment, the reinforcement learning optimization module 402 is specifically used for:

[0177] At each training step, the output of the benchmark OCR model is introduced as a structured comparison baseline, and KL divergence constraints are added.

[0178] In one embodiment, the pre-training module 401 is specifically used for:

[0179] A pre-trained model is obtained by training a visual language model using labeled image and text pairs. The visual language model includes a visual encoder and a language decoder. The visual encoder is used to encode the input image into visual features, and the language decoder is used to generate text based on the visual features.

[0180] In one embodiment, the pre-training module 401 is specifically used for:

[0181] When training the visual language model using labeled image and text pairs, the visual encoder is frozen and the language decoder is fine-tuned.

[0182] In one embodiment, the device 400 may further include:

[0183] The distillation module is used to distill a lightweight student model from the optimized model obtained by the reinforcement learning optimization module 402, and then deploy the lightweight student model on a designated terminal. The factual reward is retained during the distillation process.

[0184] Figure 5 This is a schematic diagram of a computer device in an embodiment of the present invention, such as... Figure 5As shown, this embodiment of the invention also provides a computer device 500, including a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, it implements the above-mentioned optical character recognition model reinforcement learning optimization method.

[0185] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described reinforcement learning optimization method for optical character recognition models.

[0186] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described reinforcement learning optimization method for optical character recognition models.

[0187] The method proposed in this invention enables a large-scale OCR model to maintain high accuracy and consistency in multi-scenario document recognition tasks, achieving intelligent understanding "from image to real semantic text." It can automatically determine the factual consistency between the OCR output and the original image, dynamically adjust the reward signal, and guide the model to achieve a balance between semantic accuracy, text conciseness, and correct formatting during the reinforcement learning phase. Compared with traditional supervised learning, this method significantly improves the model's factual accuracy, robustness, and interpretability, providing reliable technical support for document digitization with high accuracy requirements, such as clue extraction, case file recognition, and evidence collection.

[0188] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0189] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0190] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0191] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0192] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A reinforcement learning optimization method for an optical character recognition model, characterized in that, include: Obtain a baseline OCR model and a pre-trained model with optical character recognition capabilities; The benchmark OCR model is used as a benchmark to verify the output of the pre-trained model; wherein, the visual language model is trained using labeled image and text data to obtain the pre-trained model. A factual reward is introduced to optimize the pre-trained model using reinforcement learning, resulting in an optimized model. This factual reward includes a semantic consistency reward, a logical fact verification reward, and a production length penalty term. Semantic consistency rewards reflect the degree of consistency between the output of the pre-trained model and the labeled truth. The logical fact verification reward includes a reward for the semantic logical consistency of the pre-trained model output and a reward for verifying the structured fields in the pre-trained model output using a deterministic rule engine; the deterministic rule engine is used to determine whether the input fields meet the preset specifications. The output length penalty term reflects the deviation between the length output by the pre-trained model and the reference length, which is output by the benchmark OCR model. After obtaining the optimized model, the process also includes: using the optimized model as the teacher model for distillation to obtain a lightweight student model, deploying the lightweight student model on a designated terminal; and retaining factual rewards during the distillation process. The reward for the semantic logical consistency of the pre-trained model output is obtained as follows: The semantic logical consistency of the output of the pre-trained model is evaluated using a logic evaluator. The logic evaluator uses preset rule judgment or a lightweight neural network model to judge the logical consistency. The number of model parameters of the logic evaluator is lower than that of the pre-trained model. Alternatively, the output text of the pre-trained model can be converted into a vector, and the vector can be compared with the statement vectors in the preset business logic knowledge base using cosine similarity. Based on the comparison result, a reward is output for the semantic and logical self-consistency of the output of the pre-trained model. The statement vectors in the preset business logic knowledge base are obtained based on text with accurate logical expression. The output length penalty term is obtained according to the following formula: ； In the formula, P l As a penalty for output length, The penalty coefficient is... L 0 For reference length, L l The length of the output of the pre-trained model; Among them, based on the group relative policy optimization algorithm, factual rewards are introduced to optimize the pre-trained model through reinforcement learning, resulting in an optimized model. This includes: at each training step, the output of the benchmark OCR model is used as a structured comparison baseline, and KL divergence constraints are added. ； in, The comparison constraint loss refers to the comparison constraint term between the output of the pre-trained model and the benchmark OCR model. This represents the dynamic weight coefficient, which is used to adjust the proportion of KL divergence constraint in the total loss. Its value evolves dynamically with the training progress. y l 、y t These are the outputs of the pre-trained model and the baseline OCR model, respectively. KL ( ) indicates divergence calculation.

2. The method as described in claim 1, characterized in that, The semantic consistency reward is obtained by evaluating the consistency between the pre-trained model output and the labeled ground truth value through edit distance and semantic embedding similarity.

3. The method as described in claim 1, characterized in that, By introducing factual rewards, the pre-trained model is optimized using reinforcement learning to obtain the optimized model, which includes: Based on the group relative policy optimization algorithm, a fact reward is introduced to optimize the pre-trained model through reinforcement learning, resulting in an optimized model. The pre-trained model generates multiple candidate text sequences for each input image, calculates a fact reward for each candidate text sequence, calculates the loss function based on the fact reward, and calculates the gradient and updates the model parameters through backpropagation.

4. The method as described in claim 3, characterized in that, The calculation of the factual reward for each candidate text sequence includes: The fact reward includes semantic consistency reward and logical fact verification reward. The semantic consistency reward, logical fact verification reward and output length penalty term are fused together using the assigned weights to calculate the fact reward for each candidate text sequence.

5. The method as described in claim 1, characterized in that, The visual language model includes a visual encoder and a language decoder; the visual encoder is used to encode the input image into visual features, and the language decoder is used to generate text based on the visual features.

6. The method as described in claim 5, characterized in that, The visual language model is trained using labeled image and text data to obtain a pre-trained model, including: When training the visual language model using labeled image and text pairs, the visual encoder is frozen and the language decoder is fine-tuned.

7. A reinforcement learning optimization device for an optical character recognition model, characterized in that, include: The pre-training module is used to obtain the baseline OCR model and the pre-trained model with optical character recognition capabilities. The benchmark OCR model is used as a benchmark to verify the output of the pre-trained model; wherein, the visual language model is trained using labeled image and text data to obtain the pre-trained model. The reinforcement learning optimization module is used to introduce fact rewards and perform reinforcement learning optimization on the pre-trained model to obtain an optimized model. The fact rewards include semantic consistency rewards, logical fact verification rewards, and output length penalties. The semantic consistency reward reflects the consistency between the pre-trained model's output and the labeled truth value. The logical fact verification reward includes rewards for the semantic and logical consistency of the pre-trained model's output and for verifying the structured fields in the pre-trained model's output using a deterministic rule engine. The deterministic rule engine is used to determine whether the input fields meet preset specifications. The output length penalty reflects the deviation between the length of the pre-trained model's output and a reference length, which is output by a benchmark OCR model. The reward for the semantic logical consistency of the pre-trained model output is obtained as follows: The semantic logical consistency of the output of the pre-trained model is evaluated using a logic evaluator. The logic evaluator uses preset rule judgment or a lightweight neural network model to judge the logical consistency. The number of model parameters of the logic evaluator is lower than that of the pre-trained model. Alternatively, the output text of the pre-trained model can be converted into a vector, and the vector can be compared with the statement vectors in the preset business logic knowledge base using cosine similarity. Based on the comparison result, a reward is output for the semantic and logical self-consistency of the output of the pre-trained model. The statement vectors in the preset business logic knowledge base are obtained based on text with accurate logical expression. The output length penalty term is obtained according to the following formula: ； In the formula, P l As a penalty for output length, The penalty coefficient is... L 0 For reference length, L l The length of the output of the pre-trained model; The reinforcement learning optimization module is specifically used to: at each training step, introduce the output of the benchmark OCR model as a structured comparison baseline, and add KL divergence constraints. ； in, The comparison constraint loss refers to the comparison constraint term between the output of the pre-trained model and the benchmark OCR model. This represents the dynamic weight coefficient, which is used to adjust the proportion of KL divergence constraint in the total loss. Its value evolves dynamically with the training progress. y l 、y t These are the outputs of the pre-trained model and the baseline OCR model, respectively. KL ( ) indicates divergence calculation; Also includes: The distillation module is used to distill a lightweight student model from the optimized model obtained by the reinforcement learning optimization module, and then deploy the lightweight student model on a designated terminal. The factual reward is retained during the distillation process.

8. The apparatus as claimed in claim 7, characterized in that, The semantic consistency reward is obtained by evaluating the consistency between the pre-trained model output and the labeled ground truth value through edit distance and semantic embedding similarity.

9. The apparatus as claimed in claim 7, characterized in that, The reinforcement learning optimization module is specifically used for: Based on the group relative policy optimization algorithm, a fact reward is introduced to optimize the pre-trained model through reinforcement learning, resulting in an optimized model. The pre-trained model generates multiple candidate text sequences for each input image, calculates a fact reward for each candidate text sequence, calculates the loss function based on the fact reward, and calculates the gradient and updates the model parameters through backpropagation.

10. The apparatus as claimed in claim 9, characterized in that, The reinforcement learning optimization module is specifically used for: The fact reward includes semantic consistency reward and logical fact verification reward. The semantic consistency reward, logical fact verification reward and output length penalty term are fused together using the assigned weights to calculate the fact reward for each candidate text sequence.

11. The apparatus as claimed in claim 7, characterized in that, The visual language model includes a visual encoder and a language decoder; the visual encoder is used to encode the input image into visual features, and the language decoder is used to generate text based on the visual features.

12. The apparatus as claimed in claim 11, characterized in that, The pre-training module is specifically used for: When training the visual language model using labeled image and text pairs, the visual encoder is frozen and the language decoder is fine-tuned.

13. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.

15. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.

Images