Model training method and device

Abstract

The invention relates to a model training method and device. The method comprises the steps of obtaining a first training image set; performing first fine tuning training on the pre-trained artificial intelligence model by using the first training image set to obtain a preliminary optimization model; acquiring a second training image set; constructing a composite reward function based on the second training image set; wherein the composite reward function is used for performing multi-dimensional quantitative evaluation on the quality output by the model; and performing second fine tuning training on the preliminary optimization model based on the second training image set and the composite reward function to obtain a final optimization model. Therefore, a two-stage differential fine-tuning model strategy is realized, so that the output result of the final optimization model is highly matched with the expectation of a real scene, the problem of insufficient model practicability is fundamentally solved, and the reliability and value of the model after deployment are greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a model training method and device. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, general-purpose models based on large-scale pre-training (such as multi-modal large models and large language models) have shown strong basic capabilities in many fields. However, when these general-purpose models are directly applied to specific downstream tasks (such as document understanding, code generation, visual question answering, etc.), their performance often fails to meet the stringent requirements of practical applications. Existing model optimization schemes mainly exist in the following two paths: One path is single-stage supervised fine-tuning, which uses labeled data for a specific task to fine-tune the pre-trained model end-to-end. Although this method can make the model initially adapt to the task format, its optimization goal is usually only focused on the accuracy of content generation (such as cross-entropy loss), making it difficult to effectively guide the model to learn and output structured information (such as format, logical relationship, etc.) that is crucial for downstream applications. Another path is the design of a special model for a specific task: that is, designing and training a special model from scratch for each task. Although this method may achieve good results on a specific task, it has high development costs and the model lacks universality and scalability, making it difficult to adapt to flexible and diverse application requirements. SUMMARY

[0003] The present application provides a model training method and device to solve the technical problem of the poor quality of the output results caused by the single optimization goal of the general artificial intelligence model in adapting to specific downstream tasks, thereby causing the lack of practicality of the model in actual applications.

[0004] In a first aspect, the present application provides a model training method, which comprises: obtaining a first training image set; performing first fine-tuning training on a pre-trained artificial intelligence model using the first training image set to obtain a preliminary optimization model; obtaining a second training image set and constructing a composite reward function; wherein the composite reward function is used for multi-dimensional quantitative evaluation of the quality of the model output; performing second fine-tuning training on the preliminary optimization model based on the second training image set and the composite reward function to obtain a final optimization model.

[0005] In a possible implementation, the final optimization model is used for a text character recognition task, the composite reward function is composed of at least two of the following loss term calculation functions, and the composite reward function is obtained by weighted summation of the weight coefficients assigned to each loss term: a format loss term calculation function, a character accuracy loss term calculation function, and a syntax correctness loss term calculation function.

[0006] In a possible implementation, the calculation process of the format loss term calculation function comprises: respectively identifying boundary information of structured elements in a predicted string output by the model and in a text-labeled string; comparing the boundary information of structured elements in the predicted string and in the text-labeled string, identifying format errors existing in the predicted string, and determining types of the format errors; assigning a penalty value to each of the identified format errors according to the types of the format errors, and performing weighted summation on penalty values of all the format errors according to preset weights to obtain a format loss value.

[0007] In a possible implementation, the respectively identifying boundary information of structured elements in a predicted string output by the model and in a text-labeled string comprises: respectively scanning the predicted string output by the model and the corresponding text-labeled string based on a predefined format rule set by using a regular expression matcher, and extracting boundary information of each structured element in the predicted string and in the text-labeled string; wherein the format rule set comprises boundary identification rules for structured elements, and the boundary information comprises a starting position coordinate and an ending position coordinate of a structured element in a corresponding string.

[0008] In a possible implementation, the calculation process of the character accuracy loss term calculation function comprises: respectively converting the predicted string output by the model and the text-labeled string into sequence representations composed of multiple processing units; determining a minimum edit distance between the respective sequence representations of the predicted string and the text-labeled string; the minimum edit distance is defined as a minimum number of single-character edit operations required to convert the predicted string into the text-labeled string; mapping the minimum edit distance into a character accuracy loss value; wherein the character accuracy loss value is positively correlated with the minimum edit distance.

[0009] In a possible implementation, the calculation process of the syntax correctness loss term calculation function comprises: extracting structured elements in the predicted string output by the model; performing syntax compliance analysis on the structured elements by using a syntax parser; identifying syntax errors existing in the predicted string based on the syntax compliance analysis result, and determining types of the syntax errors; According to the type of the syntax error, a penalty value is assigned to each identified syntax error, and penalty values of all syntax errors are weighted and summed according to a preset weight to obtain a syntax correctness loss value.

[0010] In a possible implementation, the second fine-tuning training of the preliminary optimization model based on the second training image set and the composite reward function to obtain the final optimization model comprises: Inputting a sample image in the second training image set into the preliminary optimization model to generate at least two candidate output results for the sample image; For the same sample image, the composite reward function is called to calculate a total loss value of each candidate output result of the sample image pair, and a reward value corresponding to each candidate output result is determined based on the total loss value; wherein the reward value is negatively correlated with the total loss value; The generalized reward and penalty optimization algorithm is used to compare the relative differences in reward values between candidate output results, to identify the first type of candidate output result and the second type of candidate output result, and to construct a preference signal for model optimization based on the identification result; The parameter of the preliminary optimization model is updated by the back propagation algorithm based on the preference signal, aiming to maximize the probability expectation of the preliminary optimization model generating the first type of candidate output result; The above steps are repeated for iterative training until a preset iteration stopping condition is met, to obtain the final optimization model.

[0011] In a possible implementation, the first training image set comprises original document images and structured text annotations corresponding to the original document images; and the first fine-tuning training of the pre-trained artificial intelligence model based on the first training image set to obtain the preliminary optimization model comprises: The pre-trained artificial intelligence model is supervised fine-tuned based on the first training image set to obtain the preliminary optimization model.

[0012] In a possible implementation, after the final optimization model is obtained, the method further comprises: Text character recognition is performed on a to-be-recognized document image by the final optimization model to output a recognition result; Correction data of the recognition result by a user is obtained; The second training image set is updated according to the correction data and the to-be-recognized document image, and the final optimization model is incrementally trained based on the updated second training image set.

[0013] In a second aspect, the present application provides a model training device, which comprises: A first image set acquisition module is configured to acquire a first training image set; The first-stage training module is used to perform the first fine-tuning training on the pre-trained artificial intelligence model using the first training image set to obtain a preliminary optimized model. The second image set acquisition module is used to acquire the second training image set and construct a composite reward function; wherein, the composite reward function is used to perform multi-dimensional quantitative evaluation of the quality of the model output; The second-stage training module is used to perform a second fine-tuning training on the preliminary optimized model based on the second training image set and the composite reward function, so as to obtain the final optimized model.

[0014] In one possible implementation, the final optimized model is used for a text character recognition task, and the composite reward function consists of at least two of the following loss term calculation functions, wherein the composite reward function is a weighted sum by assigning weight coefficients to each loss term: Functions for calculating format loss, character accuracy loss, and syntax correctness loss.

[0015] In one possible implementation, the device further includes: The format loss term calculation module is used to identify the boundary information of structured elements in the predicted string and the text annotation string output by the model, respectively. By comparing the boundary information of structured elements in the predicted string and the text annotation string, format errors in the predicted string are identified, and the type of the format error is determined. Based on the type of the formatting error, a penalty value is assigned to each identified formatting error, and the penalty values ​​of all formatting errors are weighted and summed according to a preset weight to obtain the formatting loss value.

[0016] In one possible implementation, the format loss term calculation module identifies the boundary information of structured elements in the predicted string and the text annotation string output by the model, including: A regular expression matcher is used to scan the predicted string and the corresponding text annotation string output by the model based on a predefined set of format rules, and to extract the boundary information of each structured element in the predicted string and the text annotation string. The format rule set includes boundary identification rules for structured elements; the boundary information includes the start and end coordinates of the structured element in the corresponding string.

[0017] In one possible implementation, the device further includes: The character accuracy loss term calculation module is used to convert the predicted string and the text annotation string output by the model into sequence representations composed of multiple processing units, respectively. Determine the minimum edit distance between the sequence representations of the predicted string and the text annotation string; the minimum edit distance is defined as the minimum number of single-character edit operations required to convert the predicted string into the text annotation string. The minimum edit distance is mapped to a character accuracy loss value; wherein the character accuracy loss value is positively correlated with the minimum edit distance.

[0018] In one possible implementation, the device further includes: The syntax correctness loss term calculation module is used to extract structured elements from the predicted string output by the model; A syntax parser is used to perform syntax compliance parsing on the structured elements; Based on the results of the syntax compliance analysis, the syntax errors present in the predicted string are identified, and the types of the syntax errors are determined. Based on the type of the syntax error, a penalty value is assigned to each identified syntax error, and the penalty values ​​of all syntax errors are weighted and summed according to a preset weight to obtain the syntax correctness loss value.

[0019] In one possible implementation, the second-stage training module is specifically used for: The sample images from the second training image set are input into the preliminary optimization model to generate at least two candidate output results for the sample images; For the same image, the composite reward function is called to calculate the total loss value for each candidate output result of the image pair, and the reward value corresponding to each candidate output result is determined based on the total loss value; wherein, the reward value is negatively correlated with the total loss value; A generalized reward and penalty optimization algorithm is used to compare the relative differences in reward values ​​among candidate output results, identify the first type of candidate output results and the second type of candidate output results, and construct a preference signal for model optimization based on the identification results; With the goal of maximizing the expected probability of the model generating the first type of candidate output, the parameters of the preliminary optimized model are updated based on the preference signal using the backpropagation algorithm. Repeat the above steps for iterative training until the preset iteration stopping condition is met, and obtain the final optimized model.

[0020] In one possible implementation, the first training image set includes original document images and corresponding structured text annotations for the original document images; the first stage training module is specifically used for: The pre-trained artificial intelligence model is trained using the first training image set in a supervised manner to obtain a preliminary optimized model.

[0021] In one possible implementation, the device further includes: The model optimization module is used to perform text character recognition on the document image to be recognized after obtaining the final optimized model, and output the recognition result. Obtain the user's correction data for the recognition results; The second training image set is updated based on the corrected data and the document image to be identified, and the final optimized model is incrementally trained using the updated second training image set.

[0022] Thirdly, this application provides an electronic device, including: a processor and a memory, wherein the processor is configured to execute a model training program stored in the memory to implement the model training method described in any one of the first aspects.

[0023] Fourthly, this application provides a storage medium storing one or more programs that can be executed by one or more processors to implement the model training method described in any one aspect.

[0024] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application obtains a first training image set and uses this first training image set to perform a first fine-tuning training on a pre-trained artificial intelligence model, thereby obtaining a preliminary optimized model. Subsequently, a second training image set is further obtained, and a composite reward function is constructed. Based on the second training image set and the composite reward function, the preliminary optimized model is fine-tuned a second time to obtain the final optimized model. This solution proposes a two-stage differentiated fine-tuning model strategy, aiming to adapt to the task characteristics and model optimization needs of different training stages. In the first stage, the goal is to cultivate generalized structured recognition capabilities, injecting general and robust basic task capabilities into the model; in the second stage, the training data and the composite reward function are combined to refine and target the model output quality, effectively achieving a smooth transition and performance leap from "basic capability construction" to "excellent scene performance," making the output results of the final optimized model highly consistent with the expectations of real-world scenarios, fundamentally solving the problem of "insufficient model practicality," and significantly improving the reliability and value of the model after deployment. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0028] Figure 1 A flowchart illustrating an embodiment of a model training method provided in this application; Figure 2 A flowchart illustrating the calculation process of the format loss term calculation function provided in this application embodiment; Figure 3 A flowchart illustrating the calculation process of the character accuracy loss term calculation function provided in this application embodiment; Figure 4 A flowchart illustrating the calculation process of the syntax accuracy loss term calculation function provided in the embodiments of this application; Figure 5 A block diagram illustrating an embodiment of a model training apparatus provided in this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0031] To address the technical problem that existing general-purpose artificial intelligence models, when adapting to specific downstream tasks, suffer from poor output quality due to a singular optimization objective, thus leading to insufficient practicality in real-world applications, this application provides a model training method and apparatus. This method implements a two-stage differentiated fine-tuning strategy, ensuring that the output of the final optimized model closely matches the expectations of the real-world scenario. This fundamentally solves the problem of "insufficient model practicality" and significantly improves the reliability and value of the model after deployment.

[0032] Figure 1 This is a flowchart illustrating an embodiment of a model training method provided in this application. Figure 1 As shown, the method includes the following steps: Step 101: Obtain the first training image set.

[0033] Step 102: Use the first training image set to perform the first fine-tuning training on the pre-trained artificial intelligence model to obtain the preliminary optimized model.

[0034] Step 103: Obtain the second training image set and construct a composite reward function; wherein, the composite reward function is used to perform multi-dimensional quantitative evaluation of the quality of the model output.

[0035] Step 104: Based on the second training image set and the composite reward function, perform a second fine-tuning training on the preliminary optimized model to obtain the final optimized model.

[0036] For ease of understanding, steps 101 to 104 are explained uniformly below: The model training method provided in this application aims to significantly improve the task execution accuracy and structured output capability of artificial intelligence models when dealing with complex input data. The core of this method lies in a two-stage differentiated fine-tuning strategy. This strategy adapts to the task characteristics and model optimization needs of different training stages by designing targeted training data, model fine-tuning algorithms, and evaluation mechanisms to achieve iterative and comprehensive optimization of model performance.

[0037] Specifically, steps 101 and 102 together constitute the first stage of fine-tuning training. The core objective of the first stage of fine-tuning training is to inject the pre-trained artificial intelligence model with the basic mapping capability "from raw input data to structured output", laying the foundation for subsequent optimization.

[0038] In step 101, a first training image set is obtained. This first training image set contains two types of core data: the original input data and the structured annotations that correspond one-to-one with the original input data. Structured annotations differ from traditional annotations that only contain truth values; they not only fully describe the semantic content that the target output should possess but also integrate rich structured information. This structured information is represented using standardized format tags (e.g., using specific markup languages ​​or delimiters to identify logical units, mathematical expressions, tabular relationships, etc.). The design of the aforementioned structured text annotations, on the one hand, provides accurate and rich supervisory signals for subsequent supervised fine-tuning, enabling the model to simultaneously learn "content recognition" and "structure understanding"; on the other hand, by introducing standardized format tags (such as LaTeX strings wrapping formulas with specific delimiters), it guides the model with structured information, providing the necessary conditions for the correct execution of downstream tasks (such as parsing, rendering, etc.).

[0039] Taking text character recognition as an example, the model training method provided in this application can significantly improve the OCR recognition accuracy and structured output capability of artificial intelligence models on complex document images. To this end, the first training set of images contains two types of core data: first, original document images, covering various complex document scenarios (such as documents containing formulas, tables, and multiple layouts), ensuring the universality and representativeness of the training data; second, structured text annotations corresponding one-to-one with the original document images. Structured text annotation differs from traditional plain text annotation; it not only fully includes the text content in the document but also integrates comprehensive formatting information, such as the standardized LaTeX (Lamport TeX, a set of formats and macro packages based on the TeX typesetting system) representation of mathematical formulas, the row and column structure information of tables, and page layout information (such as heading levels, paragraph boundaries, list indentation formats, etc.).

[0040] In step 102, the pre-trained artificial intelligence model is fine-tuned for the first time based on the first training image set to obtain a preliminary optimized model.

[0041] In one embodiment, the pre-trained artificial intelligence model is fine-tuned for the first time based on the first training image set to obtain a preliminary optimized model, including: supervised fine-tuning (SFT) training of the pre-trained artificial intelligence model based on the first training image set to obtain a preliminary optimized model. Here, the specific process of supervised fine-tuning training can follow the standard supervised learning paradigm and focus on the requirements of structured recognition tasks: First, data preprocessing is performed, and the input original document image (denoted as Image_I) is standardized (including but not limited to size adjustment, pixel value normalization, etc., to adapt to the model input specifications), while the structured text label (denoted as Label_L) is converted into a token sequence that the model can recognize; then, model forward propagation is carried out, and the preprocessed Image_I is input into the pre-trained artificial intelligence model, which jointly recognizes and encodes the text content and format information in Image_I; next, loss calculation is performed, using cross-entropy loss as the optimization objective to quantify the difference in probability distribution between the predicted token sequence output by the model and the token sequence of the structured text label Label_L; finally, through backpropagation and parameter update, based on the calculated cross-entropy loss, the model parameters are iteratively updated using the gradient descent algorithm (which can be combined with optimizers such as AdamW (Adam with Weight Decay, Adam optimizer with weight decay)) to gradually minimize the loss function value.

[0042] Through the first stage of fine-tuning training described above, the final output is a preliminary optimized model (denoted as Model_SFT). This preliminary optimized model, under the supervision of the first training image set, has mastered the basic mapping rules from raw input data (such as document images) to structured output (such as structured text), and has the preliminary ability to recognize content information, such as text characters, mathematical formulas, and table content. It can also output structured output containing formatting tags, providing a qualified basic model for the second stage of performance optimization.

[0043] Steps 103 and 104 constitute the second stage of fine-tuning training. The core objective of the second stage of fine-tuning training is to address the problems that the preliminary optimized model obtained from the first stage of fine-tuning training may have, such as "insufficient adaptability to real application scenarios" and "lack of multi-dimensional quantitative optimization of output text quality," and to further improve the model's task performance in real user scenarios.

[0044] Step 103 includes two aspects: First, acquiring a second training image set, which differs from the general data in the first stage. This second training image set focuses on collecting sample data based on "actual user application scenarios," and each sample data is equipped with structured annotations consistent with the specifications of the first stage. This aims to make the training data closer to real-world usage scenarios, ensuring that the optimized model can adapt to the data characteristics in real-world scenarios, thereby ensuring that the model's output content (such as mathematical formulas and table structures) can be correctly recognized, parsed, and presented by downstream processing systems (such as document rendering engines and layout restoration tools). Second, designing a composite reward function, which breaks through the traditional single-dimensional evaluation logic and uses multi-dimensional indicators to quantitatively score the quality of the model output.

[0045] Continuing with the text recognition task as an example, the second training image set includes sample images collected with "user's actual application scenarios" as the core (such as contract documents in corporate office scenarios, courseware documents in educational scenarios, paper documents in scientific research scenarios, financial statements in financial scenarios, etc.). Each sample image is equipped with structured text annotations consistent with the specifications of the first stage to ensure that the output text of the final optimized model is a structured string containing format marks, thereby ensuring that the model output content (such as mathematical formulas and table structures) can be correctly recognized, parsed and presented by downstream processing systems (such as document rendering engines and layout restoration tools).

[0046] In text recognition tasks, for example, a composite reward function can cover core dimensions such as recognition accuracy (the degree of matching between text content and annotation), format restoration (the consistency between output format and original document), and readability (the logical coherence of structured strings), providing clear optimization directions and reward signals for subsequent reinforcement learning.

[0047] In step 104, the preliminary optimized model (Model_SFT) is fine-tuned a second time based on the second training image set and the composite reward function to obtain the final optimized model.

[0048] In one embodiment, the preliminary optimized model is fine-tuned a second time based on a second training image set and a composite reward function to obtain the final optimized model. This includes: fine-tuning the preliminary optimized model based on reinforcement learning using the second training image set and the composite reward function to obtain the final optimized model. Specifically, sample images from the second training image set are used as input, and Model_SFT outputs structured text results. The output results are evaluated in multiple dimensions using the composite reward function to generate corresponding reward values. With the goal of "maximizing the reward value," reinforcement learning algorithms (such as PPO (proximal policy optimization) algorithms) are used to iteratively update the model parameters, enabling the model to gradually learn output strategies that meet the needs of the actual scenario and are superior in multiple quality metrics.

[0049] Through the second stage of fine-tuning training, the final optimized model is obtained (e.g., a final optimized model for text character recognition tasks). This final optimized model inherits the basic structured recognition capabilities of the first stage and, through optimization using real-world scenario data and a multi-dimensional reward mechanism, significantly improves both the recognition accuracy for complex data and its adaptability to real-world user application scenarios. For example, the final optimized model for text character recognition tasks obtained by applying the technical solution provided in this application can significantly improve the OCR recognition accuracy of complex document images (especially the recognition accuracy of complex elements such as formulas and tables), the format restoration accuracy of structured output, and the adaptability to real-world user application scenarios. It can be directly applied to OCR-related business scenarios that require high-precision structured text output.

[0050] The technical solution provided in this application obtains a first training image set and uses it to perform a first fine-tuning training on a pre-trained artificial intelligence model, thereby obtaining a preliminary optimized model. Subsequently, a second training image set is obtained, and a composite reward function is constructed. Based on the second training image set and the composite reward function, the preliminary optimized model is subjected to a second fine-tuning training to obtain the final optimized model. This solution proposes a two-stage differentiated fine-tuning model strategy, aiming to adapt to the task characteristics and model optimization needs of different training stages. In the first stage, the goal is to cultivate generalized structured recognition capabilities, injecting the model with general and robust basic task capabilities. In the second stage, the training data and the composite reward function are combined to refine and target the model output quality, effectively achieving a smooth transition and performance leap from "basic capability construction" to "excellent scene performance." This ensures that the output results of the final optimized model highly match the expectations of real-world scenarios, fundamentally solving the problem of "insufficient model practicality" and significantly improving the reliability and value of the model after deployment.

[0051] Continuing with the text recognition task as an example, in one embodiment, the aforementioned composite reward function consists of at least two of the following loss term calculation functions: a format loss term calculation function, a character accuracy loss term calculation function, and a grammatical correctness loss term calculation function. The composite reward function is a weighted sum of each loss term, with weight coefficients assigned to them. For example, the composite reward function is shown in the following formula (I): Formula (1) in, The function that calculates the format loss term. The function that calculates the character accuracy loss term. This represents the function for calculating the loss term for grammatical correctness. , , These are the weighting coefficients. For example, the weighting coefficients for each loss term are preset empirical values.

[0052] The format loss term calculation function is a quantification module in the composite reward function that targets the "accuracy of the structured text format of the model output". Its core objective is to accurately identify and quantify format errors in the model's recognition results (such as format problems of complex structured elements such as formulas and tables) by comparing and evaluating the format dimension of the predicted strings output by the model with the labeled structured text strings. This provides a clear format optimization guide for the optimization of model parameters and ultimately ensures that the structured strings output by the model (such as mathematical formulas and table structures) can be correctly recognized, parsed and presented by downstream processing systems (such as document rendering engines and layout restoration tools), avoiding downstream processing failures caused by non-standard formats.

[0053] In one embodiment, see Figure 2 The process shown includes the following steps in calculating the format loss term calculation function: Step 201: Identify the boundary information of structured elements in the predicted string and the text annotation string output by the model, respectively.

[0054] The core task of step 201 is to accurately extract the boundary information of all structured elements from the predicted string output by the model and the text annotation string used as the benchmark, respectively, so as to provide a data foundation for subsequent identification of format errors and quantification of format loss by comparison.

[0055] Structured elements refer to content in a document that needs to be defined using formatting tags, as opposed to plain text that does not require formatting tags. Typical examples include mathematical formulas, tables, and lists (which require formatting tags). ... " ...Elements with formatting attributes, such as indentation level markers (e.g., headings, code blocks), and titles, directly determine whether downstream systems can parse them correctly. If formatting markers are missing or incorrect, downstream systems may fail, for example, recognizing formulas as plain text or breaking tables into unordered characters, causing the core goal of the OCR task, "structured output," to fail.

[0056] Furthermore, the primary prerequisite for the correct format of structured elements is that "boundary markers conform to standards." Boundary information (including the start and end coordinates of the element within its corresponding string) is the core basis for judging the completeness and accuracy of boundary markers. Only by clearly defining the start and end range of a structured element can the format markers enclosing that structured element be further verified to be standardized. Based on this, step 201 identifies the boundary information of structured elements in the predicted string and the text annotation string output by the model, respectively.

[0057] In one embodiment, an exemplary implementation of identifying the boundary information of structured elements in the predicted string and the text annotation string output by the model includes: using a regular expression matcher to scan the predicted string and the corresponding text annotation string output by the model based on a predefined set of format rules, and extracting the boundary information of each structured element in the predicted string and the text annotation string.

[0058] The formatting rules set includes boundary identification rules for various structured elements. These boundary identification rules are formulated based on industry-standard criteria that can be parsed by downstream processing systems, clearly defining the boundary symbols and syntax specifications of different structured elements, ensuring that the extracted boundary information can directly adapt to the format parsing requirements in practical applications. For example, for mathematical formulas, the formatting rules set includes boundary identifiers for two mainstream formats: one is the formula boundary identifier under LaTeX format, such as "$...$" and "\(...\)"; the other is the formula boundary identifier under MathML (Mathematical Markup Language) format, such as "$...$" and "\(...\)". <m> ...< / m> For tables, the formatting rule set contains boundary markers that define the table's row and column structure, such as "". ... "" indicates the boundary of the entire table, while "..." indicates the boundary of individual cells within the table. For list content, the formatting rules set includes boundary markers that distinguish list types: the overall boundary marker for an unordered list is "". ... The boundary marker for list items is "". ... The overall boundary of an ordered list is marked as "". ... ".

[0059] The regular expression matcher is invoked to perform a full scan of the model's output predicted string and corresponding text annotation string according to the above format rule set, matching predefined boundary identifiers (such as "$"). <m>and the text-labeled string. Here, the regular expression matcher is selected as the tool because it has high specific pattern matching capability and can quickly identify the boundary markers in the string that meet the format rules, thereby efficiently locating the start and end ranges of the structured elements. It is particularly suitable for batch string format parsing requirements, can balance processing efficiency while ensuring extraction accuracy, and meets the performance requirements for format evaluation of large amounts of data during model training.

[0060] Step 202: Compare the boundary information of the structured elements in the predicted string and the text-labeled string to identify format errors in the predicted string and determine the type of format error.

[0061] Step 202 is based on the boundary information extracted in step 201, and by comparing the boundary information of the structured elements in the predicted string output by the model and the text-labeled string as the reference, the format errors in the predicted string are accurately identified and classified, providing clear error basis for subsequent quantitative format loss.

[0062] Specifically, in step 202, a one-to-one correspondence between the structured elements in the two types of strings is first established. Since the predicted string output by the model may have missing structured elements, sequence deviation, or repeated identification, etc., by "semantic matching" (comparing the core content of the elements, such as the mathematical expression of the formula, the cell data of the table) or "position sequence matching" (according to the order of the elements appearing in the string), the structured elements representing the same content in the two types of strings are locked, ensuring the effectiveness of the comparison objects and avoiding incorrect judgments due to element misplacement.

[0063] After establishing the correspondence, for each pair of matched structured elements, the boundary information of the two is compared in multiple dimensions: whether the boundary marker symbols meet the pre-defined format rule set (such as whether the LaTeX formula uses the "$...$" marker, whether the MathML formula uses the <m> ...< / m> " marker), whether the start position coordinates are consistent, whether the end position coordinates are consistent, and whether the boundary markers are complete (such as only the start marker exists without the end marker, or vice versa).

[0064] By the above comparison, the format problems of the structured elements in the predicted string can be identified. For example: the formula is only started with "$" but not ended with "$" (missing boundary identifier), the formula is started with "$" but ended with "\" (boundary identifier mismatch), the LaTeX formula mark "$...$" is mistakenly written as "#...#" (boundary identifier symbol error), the character index of the starting mark is 3 characters off the mark (boundary position offset), and the like.

[0065] On this basis, the format errors are classified according to the characteristics of the format errors, and a format error type system is formed. Exemplarily, the classification logic takes "the influence degree of the error on the analysis of the downstream system" and "the technical essence of the error" as the double basis, for example, the format error types include: Boundary missing and mismatching type, which refers to one of the following two problems in the corresponding position of the predicted string in the region marked as a formula in the text annotation string: one is that the starting delimiter or the ending delimiter is completely missing, and no valid boundary identifier is detected; the other is that the starting delimiter exists, but the corresponding ending delimiter is missing or the ending delimiter type is wrong (such as starting with LaTeX format "$", ending with "\"). This type of error will cause the downstream system to be unable to accurately define the range of structured elements such as formulas, and belongs to the error that seriously affects the effectiveness of analysis; Identifier type error type: which refers to the use of other types of delimiters defined in the format rule set (rather than non-standard symbols outside the rule set) in the corresponding position of the predicted string in the region marked as a formula in the text annotation string. For example, the LaTeX inline formula delimiter "$...$" should be used, but the same rule set inside <m> ...< / m> ". This type of error will cause the downstream system to deviate in element format identification (such as misjudging inline formulas as other formats), affecting the accuracy of structured analysis; Extraneous identifier type: which refers to the erroneous insertion of delimiters in the format rule set (such as mistakenly adding "$" in pure text <m>"etc.). Such errors can cause downstream systems to misidentify unstructured elements as structured elements such as formulas, interfering with the normal parsing process and affecting the overall layout restoration of the document; Duplicate identification class: refers to the same structured element being repeatedly added with boundary identifiers (such as a formula being double-wrapped with "$$...$$", a table being repeatedly marked with ... ... ..."). Such errors can cause redundant judgments when parsed by downstream systems, such as repeatedly rendering the same element, affecting the accuracy of format restoration and the display effect of the document.

[0066] Through the above format error identification and classification, step 202 finally outputs a format error set containing the corresponding structured element and error type, thereby providing guidance for subsequent optimization of model format output capabilities through a penalty mechanism.

[0067] Take the identification process of the formula "start delimiter missing" error as an example: In the text annotation string, the original content is: "The equation $E=mc^2$ is famous.".

[0068] The structured element (math formula) "E=mc^2" is wrapped with LaTeX inline formula standard delimiters "$...$", forming a complete structured identifier that meets the format rule set, clearly defining the scope of the formula.

[0069] The predicted string output by the model is: "The equation E=mc^2$ is famous.".

[0070] For the content in the predicted string corresponding to the formula region of the text annotation string (i.e., "E=mc^2" at character indices 12-19), delimiter detection is performed according to the format rule set, and the result shows that only the end position of the region detects the formula delimiter, while the start position (corresponding to the position of the start delimiter "$" in the text annotation string) does not detect the valid delimiter defined by the rule set (actually the character "E", which is not a LaTeX formula boundary identifier).

[0071] Based on the above detection results, it can be clearly identified that this format error belongs to the "start delimiter missing" error in the "boundary missing and mismatch" class. Since the predicted string lacks the start delimiter of the formula, the downstream system will not be able to accurately determine the start boundary of the formula, which may cause "E=mc^2" to be misidentified as ordinary text (or determined as an incomplete formula), affecting the accuracy of structured parsing.

[0072] Step 203, assign a penalty value to each identified format error according to the type of format error, and weight sum all the penalty values of the format errors according to the preset weight to obtain a format loss value.

[0073] The core task of step 203 is to convert the "format error" identified in step 202 into a "quantifiable format loss value", providing a clear numerical guidance for model optimization. The size of the format loss value intuitively reflects the format specification of the structured text output by the model, thereby driving the model to reduce the format error rate through parameter adjustment, ultimately meeting the requirements of the downstream system for format accuracy.

[0074] In an embodiment, a "format error type-penalty value" mapping table can be pre-constructed, where the penalty value is set in association with the degree of impact of the error on the parsing of the downstream system. For example, "boundary missing type" errors (such as missing end marks for formulas) can cause the downstream system to completely fail to define the element range, which is a serious error and can correspond to a higher penalty value (such as 10.0); "identification error type" errors (such as replacing LaTeX standard marks with non-standard symbols) can cause the downstream system to fail to identify the meaning of the marks, and a higher base penalty value (such as 8.0) is also set; "repeated identification type" errors (such as double marking of formulas) can cause redundant parsing, and the penalty value can be set to a medium level (such as 5.0).

[0075] Subsequently, in combination with the "tolerance requirements" of various types of format errors in actual application scenarios, preset weights are configured for different format error types. For example, if the business scenario (such as scientific paper OCR) has very high requirements for formula format integrity, the weight of "boundary missing type" errors can be increased to 1.5; if the position accuracy requirement of the table format is low, the weight of "position offset type" errors can be reduced to 0.5, and the key error types are optimized through weight adjustment.

[0076] After completing the penalty value and weight configuration, the format error set output by step 202 is traversed, and the corresponding penalty value is matched for each format error according to the format error type and multiplied by the preset weight of the type to which it belongs, to obtain the weighted penalty value of a single format error. Finally, the weighted penalty values of all format errors are summed to obtain the final format loss value.

[0077] Therefore, the format loss value is the core quantitative indicator of the "format dimension" in the composite reward function. The higher the format loss value, the worse the structured text format specification of the model output, and the model parameters related to format recognition and label generation need to be adjusted. Conversely, the lower the format loss value, the more the format output conforms to the standard. Through this quantitative process, the model can clearly perceive the direction of format optimization, avoid the problem of rough optimization granularity caused by "only qualitative judgment of right or wrong", and ultimately achieve the technical goal of improving the accuracy of structured output format and ensuring the correct parsing of downstream systems.

[0078] The character accuracy loss term calculation function is a quantitative module in the composite reward function for "consistency of characters in the model output predicted string and the text annotated string". Its core goal is to accurately measure the differences in characters (including structured element format label characters and pure text characters) between the two types of strings, provide a clear quantitative guide for "character content recognition accuracy" for model optimization, ensure that the model prioritizes the accuracy of core text information while learning structured output format, and avoid errors in structured text information caused by character errors, omissions, and misplacements (such as writing "^2" as "2^"), which can affect the correct understanding of text semantics by downstream systems.

[0079] In an embodiment, referring to the flowchart shown in Figure 3 , the calculation process of the character accuracy loss term calculation function includes the following steps: Step 301, convert the predicted string output by the model and the text annotated string into a sequence representation composed of multiple processing units.

[0080] The core task of step 301 is to uniformly convert the two types of strings into a sequence representation composed of multiple processing units that the model can calculate, such as token sequences, to lay the foundation for "same dimension, same standard" for subsequent character-level difference comparison.

[0081] Specifically, first, load the text tokenizer consistent with the model training to ensure that the predicted string and the text annotated string use the same token mapping rules, including the unified processing of ordinary text characters (such as letters, numbers, and punctuation) and structured element format label characters (such as LaTeX's " $...$ ", and table's " |...| "). ... Among them, the text tokenizer is a word divider, and its core function is to divide a string into a sequence of discrete processing units (such as characters, subwords, or complete words) according to a predefined vocabulary and division rule, and map each unit to a unique numerical identifier, thereby converting text data into a numerical sequence representation that the model can process.

[0082] Subsequently, operations are performed on the two types of strings respectively: the strings are disassembled into discrete tokens according to the splitting rules (for example, E=mc^2 can be split into "E" "=" "m" "c" "^2" tokens corresponding to the tokens), and converted into token sequences recognizable by the model. The key value of this step is to convert the variable-length string into a fixed-length or variable-length ordered token sequence, so that the subsequent character difference calculation can be realized by token sequence alignment, while the token information of the format marking character of the structured element is completely preserved, avoiding the omission of the accuracy evaluation due to the tokenization of the format marking character (such as <m>"miswritten as" <n>The difference between the two can be captured by token comparison).

[0083] Step 302, determining the minimum edit distance between the respective sequence representations of the predicted string and the text-labeled string; the minimum edit distance is defined as the minimum number of single-character edit operations required to convert the predicted string into the text-labeled string.

[0084] Step 302 realizes the overall quantification of the character difference between the two types of strings by calculating the minimum edit distance between the two types of sequence representations (such as token sequences).

[0085] Among them, the core definition of the minimum edit distance is: under the unified token sequence alignment rule, the minimum number of single token edit operations required to convert Token_pred (token sequence of the predicted string) into Token_label (token sequence of the text-labeled string), and the supported edit operation types include three types: token insertion (adding a token in Token_pred to match Token_label), token deletion (removing a redundant token in Token_pred), and token replacement (replacing a token in Token_pred with the token at the corresponding position in Token_label).

[0086] For example, if Token_label is "$”"E”"=”"m”"c”"^2”"$", and Token_pred is "E”"=”"m”"c”"2”"$". It can be seen that Token_pred has two key deviations: one is the missing starting marker "$" of the formula, and the other is the misrecognition of the formula core character token "^2" as "2".

[0087] When calculating the minimum edit distance, in order to make Token_pred match the starting format marker of Token_label, a "token insertion" operation needs to be performed at the beginning of Token_pred ("E" before) to insert the missing starting marker token "$"; in order to correct the recognition deviation of the formula core character, the token "2" in Token_pred needs to be replaced with the correct token "^2" by performing a "token replacement" operation. Then, it takes 2 minimum edit operations (1 insertion and 1 replacement) to convert Token_pred into Token_label, so the minimum edit distance between the two is 2.

[0088] Step 303, mapping the minimum edit distance to a character accuracy loss value; wherein the character accuracy loss value is positively correlated with the edit distance.

[0089] Step 303 converts the "minimum edit distance" into a "character accuracy loss value", providing a quantitative indicator for model parameter optimization.

[0090] In an embodiment, the mapping rule of the "minimum edit distance-loss value" is preset based on the task scenario of model training, such as the tolerance of character accuracy in document OCR. The core logic is that the "character accuracy loss value is positively correlated with the minimum edit distance", that is, the greater the minimum edit distance, the more significant the character difference between the predicted string and the labeled string, and the higher the character accuracy loss value; otherwise, the loss value is lower.

[0091] Among them, the mapping method can adopt one of the following two methods: one is linear mapping, which introduces a preset coefficient to make the character accuracy loss value = preset coefficient x minimum edit distance; the second is nonlinear mapping (such as logarithmic mapping, segmented mapping), when the edit distance is small, linear mapping is adopted, when the edit distance exceeds a preset threshold (such as 10), logarithmic scaling is adopted to avoid loss value overflow caused by extreme large edit distance, affecting the stability of model training.

[0092] By converting the "minimum edit distance" into a "character accuracy loss value", the model can adjust the parameters related to "character recognition" (such as the mapping parameters of image features to text tokens in a multi-modal model) based on the gradient signal of the character accuracy loss value through the back propagation algorithm, gradually reduce the minimum edit distance, and ultimately improve the accuracy of character content recognition.

[0093] The syntax accuracy loss term calculation function is a quantitative module in the composite reward function for the "syntax compliance of structured elements (such as mathematical formulas, tables, lists, etc.) in the model output predicted string". Its core goal is to strictly check whether the syntax within the structured element conforms to the corresponding format standard (such as LaTeX formula syntax, HTML table syntax, MathML semantic syntax, etc.), accurately identify and quantify syntax-level errors, provide a quantitative guide for "structured element semantic analyzability" for model optimization, and ensure that the structured elements output by the model not only have complete format markers and accurate character content, but also pass the syntax compliance verification, avoiding the problem of "formally complete but functionally ineffective" caused by syntax errors (such as missing operator symbols in formulas, table tags not closed, list level nesting errors) that prevent downstream systems from correctly parsing element semantics, and thus causing structured output.

[0094] In an embodiment, referring to the flowchart shown in Figure 4 The calculation process of the syntax accuracy loss term calculation function includes the following steps: Step 401, extract the structured elements in the predicted string output by the model.

[0095] Since syntax errors only exist within structured elements with specific format standards, plain text usually has no strong constraints on "syntax compliance", so the core task of step 401 is to separate structured elements that need to be checked for syntax from the predicted string output by the model, laying the foundation for subsequent targeted syntax parsing.

[0096] In an embodiment, based on the format rule set defined above, a regular expression matcher or structured element extractor is called to scan the predicted string for content that matches the characteristics of the format markers (such as LaTeX formulas wrapped in "$...$", tables wrapped in " ... ", MathML formulas wrapped in " <m> ...< / m> "), thereby extracting structured elements from the predicted string.

[0097] Step 402, using a syntax parser to parse the syntax compliance of the structured elements.

[0098] In an implementation, step 402 checks the syntax compliance of each structured element in the predicted string by adapting a dedicated syntax parser for different structured elements, the core being to ensure that the parsing logic is completely aligned with the syntax standards of the downstream system. Specifically, first, according to the type of structured element (such as LaTeX formula, HTML table, MathML formula), the corresponding dedicated syntax parser is called, for example, for LaTeX formula elements, a syntax parser supporting LaTeX 2ε (pronounced LaTeX Two-Epsilon, which is the most widely used version of LaTeX) standard is called; for HTML table elements, a table syntax parser that conforms to W3C (World Wide Web Consortium) HTML5 (HyperText Markup Language 5) standard is called; for MathML formula elements, a syntax parser compatible with MathML3.0 standard is called. Subsequently, the syntax parser performs the full process of "lexical analysis → syntax analysis → semantic verification" on each structured element.

[0099] Among them, lexical analysis is to split the element content into syntax units (such as "$" "E" "=" "mc^2" in LaTeX formula, " ”" ”" "), check whether the unit conforms to the format standard (such as whether there are undefined special symbols); Syntax analysis is based on a syntax rule tree (such as the precedence rules of LaTeX formulas and the nesting rules of table tags) to verify whether the combinational logic of syntax units is compliant (such as whether brackets in LaTeX formulas are paired, and whether nesting rules of table tags are correct). "whether nested in"

[0100]

[0101]

[0102]

[0103] internal); semantic checking is for elements with semantic logic (such as the operation relationship of the formula, the row-column correlation of the table), and checks whether the syntax structure can support normal semantic expression (such as whether the superscript syntax of "^" in the formula "E=mc^2" is correct, to avoid semantic distortion caused by "E=mc2^"). After parsing, the syntax compliance parsing result of each structured element is output, and it is explicitly marked as "passed the check" or "there is a syntax error", which provides a basis for subsequent error identification. Step 403, based on the syntax compliance parsing result, identifies the syntax errors existing in the predicted string and determines the type of the syntax errors. Step 403, based on the syntax compliance parsing result of step 402, locates and classifies the structured elements with syntax errors in the predicted string. Specifically, first, for each structured element marked as "there is a syntax error", the error log output by the syntax parser (such as "LaTeX formula missing right parenthesis" "HTML table 'tr' tag missing 'td' tag" and the like) is combined to determine the type of the syntax error. For example, if the error log is "LaTeX formula missing right parenthesis", it can be determined that the type of the syntax error is "missing right parenthesis". If the error log is "HTML table 'tr' tag missing 'td' tag", it can be determined that the type of the syntax error is "missing 'td' tag". The specific types of syntax errors are not limited to the above examples, and can be determined according to the error log output by the syntax parser. "tag not closed" ), pinpointing the specific location in the element (e.g. the missing right parenthesis in "(a+b" in a formula, the row index of the unclosed tag in a table); then according to the technical features of the error, the degree of impact of the error on the downstream system parsing, and the type of violation of the syntax rules, the error is classified into preset types, including the following error types: Syntax structure missing type: such as missing key operator symbols in a LaTeX formula (e.g. "a+b" is mistakenly written as "ab"), missing " "close tag, missing MathML formula, semantic container tag, such errors will cause downstream systems unable to build a complete syntax tree, is a serious error; Misuse of syntax rules: such as LaTeX formula superscript symbol "^" used after non-character (such as "a^+b" miswritten as "^a+b"), command spelling error (such as "\alfha" because of the letter spelling error leads to undefined, the correct should be "\alpha"), HTML table " " tag nested in " " inside (violate the level rules), MathML formula operator + "not placed in" Inside, such errors will cause semantic deviation in downstream system parsing; Syntax redundancy class: such as the redundant parentheses in LaTeX formula ("(a+b)" is miswritten as "((a+b))"), the redundant right bracket after the square root command (such as "\sqrt{x} " caused by the extra "}"), the redundant "border=0" attribute in HTML table (although it does not affect parsing but does not conform to the simplicity specification), such errors have little effect on core semantic parsing, which are minor errors.

[0104] Step 404, according to the type of syntax error, assign a penalty value to each identified syntax error, and weight sum all the penalty values of syntax errors according to the preset weight, get the syntax correctness loss value.

[0105] The core task of step 404 is to convert the "syntax error" identified in step 403 into "syntax correctness loss value", which provides a clear numerical guidance for model optimization. The size of the syntax correctness loss value intuitively reflects the syntax correctness of the structured text output by the model, and then drives the model to reduce the syntax error rate through parameter adjustment, and finally meets the requirements of the downstream system for syntax correctness.

[0106] In an embodiment, a "syntax error type-penalty value" mapping table is constructed in advance, wherein the setting of the penalty value is related to the degree of influence of the error on the parsing of the downstream system. For example, the "syntax structure missing class" error causes parsing failure, and a high penalty value (such as 15.0) is set; the "syntax rule misuse class" error causes semantic deviation, and a medium penalty value (such as 8.0) is set; the "syntax redundancy class" error has little effect, and a low penalty value (such as 2.0) is set.

[0107] Subsequently, according to the requirements of the syntax accuracy in actual application scenarios (such as the high requirement of scientific research paper OCR on formula syntax, and the high tolerance of ordinary office documents on table syntax), weights are configured for different error types. For example, the weight of "syntax structure missing class" error is set to 2.0 in scientific research scenarios, and 1.2 in ordinary office scenarios.

[0108] After completing the penalty value and weight configuration, each syntax error output by step 403 is traversed, and the penalty value is matched according to its type and multiplied by the corresponding weight to get the weighted penalty value of a single syntax error; the weighted penalty values of all syntax errors are summed up, and the syntax correctness loss value is obtained.

[0109] Therefore, the syntax correctness loss value is the core indicator of "semantic analyzability" in the composite reward function. The higher the syntax correctness loss value, the worse the syntax compliance of the structured element, and the model needs to focus on adjusting the parameters related to "structured element syntax generation" (such as the network layer parameters responsible for format marking and content fusion in the multi-modal model). Conversely, the lower the loss value, the more the syntax output conforms to the standard. Through this quantitative process, the model can further achieve "syntax compliance" on the basis of "format integrity" and "character accuracy".

[0110] In an embodiment, in the second-stage fine-tuning training, based on the second training image set and the composite reward function, the preliminary optimization model is fine-tuned based on reinforcement learning to obtain the final optimization model. An exemplary implementation includes: inputting a sample image in the second training image set into the preliminary optimization model to generate at least two candidate output results for the sample image; for the same sample image, calling the composite reward function to calculate the total loss value of each candidate output result pair of the sample image, and determining the reward value corresponding to each candidate output result based on the total loss value; wherein the reward value is negatively correlated with the total loss value; using a generalized reward and penalty optimization algorithm to compare the relative differences in reward values between the candidate output results, identify the first type of candidate output result and the second type of candidate output result, and construct a preference signal for model optimization based on the identification result; based on the preference signal, update the parameters of the preliminary optimization model by a back propagation algorithm to maximize the probability expectation of generating the first type of candidate output result; repeat the above steps for iterative training until a preset iteration stopping condition is met, and obtain the final optimization model.

[0111] The above embodiments describe how to iteratively optimize the preliminary optimization model into a final model suitable for text character recognition tasks based on reinforcement learning algorithms, combined with the second training image set and the composite reward function in the second-stage fine-tuning. The core logic is to use a closed loop of "multiple candidate output comparison-reward signal quantization-preference guided optimization" to enable the model to learn "a better structured output strategy" on sample data in actual user scenarios, and ultimately improve the format specification, character accuracy and syntax compliance of the model in real applications.

[0112] Specifically, first, input a single sample image (such as a user-uploaded scanned image of an office document containing formulas and tables) in the second training image set into the preliminary optimization model, and generate at least two candidate output results (denoted as C1, C2,..., Cn) through "random sampling" or "diversified generation strategy" (such as introducing a temperature coefficient to adjust randomness in the output layer of the model). n , n≥2). The core purpose of generating multiple candidate outputs is to increase the "exploration" of model output, avoid the model being limited to a single output mode, and thus discover better structured expressions (such as there may be two compliant formats of "$E=mc^2$" and "\ (E=mc^2)", and through multiple candidate comparison, the format selection suitable for the scene can be learned); provide a sample basis for subsequent "reward difference comparison". Only when there are multiple output options can the reward value be used to distinguish between good and bad, and thus build a preference signal for model optimization.

[0113] Next, based on the composite reward function, the output quality is quantified: for all candidate output results of the same image, the pre-constructed composite reward function is called to calculate the total loss value of each candidate output (see formula (I) above), and then the total loss value is mapped to the "reward value". The mapping rule is that the reward value is negatively correlated with the total loss value, that is, the smaller the total loss value (the higher the output quality), the higher the reward value; the larger the total loss (the lower the output quality), the lower the reward value.

[0114] Next, identify the candidate type and build the preference signal: the model optimization direction is clear, and the "generalized penalty optimization algorithm" (a reinforcement learning strategy based on reward relative difference, similar to PPO but more focused on candidate comparison) is used to compare or globally sort the reward values of all candidate output results of the same image, and identify two types of candidate results: the first type of candidate output result: the reward value is higher (such as the top 30%), representing a better structured output (such as complete format, accurate characters, and grammatically correct); the second type of candidate output result: the reward value is lower (such as the last 70%), representing a poor structured output (such as format errors, character omissions, and syntax problems). Based on this classification, a "preference signal" is built, that is, a mathematical way is used to express that "the model should be more inclined to generate the first type of candidate, rather than the second type of candidate", for example, by calculating the preference difference value of "the reward value of the first type of candidate - the reward value of the second type of candidate", the larger the difference value, the stronger the signal, indicating that the model needs to be optimized more significantly to favor the first type of output.

[0115] In the next step, the model parameters are updated: the output strategy is optimized based on the preference signal to maximize the probability expectation of the model generating the first type of candidate output result, and the constructed preference signal is used as the optimization basis to adjust the parameters of the preliminary optimization model through the back propagation algorithm. Specifically, the gradient direction of the model parameter update is determined by the preference signal: if the preference signal of the first type of candidate is stronger than that of the second type of candidate, the parameters related to the generation of the first type of candidate features in the model (such as the mapping weight of image features to LaTeX marks and the network layer parameters generated by the grammar rule) will be strengthened, while the parameters related to the generation of the second type of candidate features will be weakened. In addition, to avoid the instability of the model caused by the excessive updating of the parameters, a "strategy constraint" (such as limiting the KL divergence (Kullback-Leibler Divergence) between the new parameters and the old parameters) can be introduced to ensure that the model retains the basic ability learned in the early SFT stage while learning a better strategy.

[0116] Finally, the gradual convergence of the model performance is achieved by repeating the above process, i.e., performing multiple rounds of iterative training on all sample images in the second training image set until the preset iteration stopping condition is met, and terminating the training.

[0117] The iteration stopping condition is that the total loss value of the validation set decreases by less than a threshold (such as 0.01) for consecutive N rounds (such as 10 rounds), indicating that the model performance tends to be stable; or the number of iterations reaches a preset upper limit (such as 50 rounds) to avoid overfitting; or the model reaches a preset target (such as a format error rate of less than 5%) on a key indicator (such as the format error rate or the character accuracy).

[0118] The above embodiment uses the "trial and error-reward-optimization" mechanism of reinforcement learning to enable the model to learn a better structured output strategy on user real scene data, which makes up for the limitations of the first stage of supervised fine-tuning relying only on fixed labeled data. Compared with traditional fine-tuning, it has the following advantages: 1. It does not rely on "absolute correct labeling", but learns "relatively better" through candidate comparison, which is more suitable for the "multiple solution compliance" situation in real scenarios (such as multiple compliant formats for the same formula); 2. The multi-dimensional evaluation of the composite reward function ensures that the model optimizes the format, character, and syntax at the same time, avoiding the "trade-off" caused by single indicator optimization; 3. The preference signal guided parameter update enables the model to improve high-frequency errors (such as the common problem of missing table labels in user scenarios), and ultimately outputs structured text that better meets the needs of practical applications.

[0119] Finally, the application also provides the following embodiment: after obtaining the final optimization model, performing text character recognition on a to-be-recognized document image using the final optimization model, and outputting a recognition result; obtaining correction data of the recognition result from a user; updating the second training image set according to the correction data and the to-be-recognized document image, and performing incremental training on the final optimization model using the updated second training image set.

[0120] The above embodiment describes a technical solution for continuously iterating the performance of a model after obtaining a final optimization model for text character recognition through a closed-loop process of "actual application-user feedback-data update-incremental training". The core logic is to use user correction data in real scenarios to make up for possible scene coverage deficiencies and poor adaptation to special cases in the model training stage, so that the model continuously adapts to user needs in actual application, further improving recognition accuracy and generalization ability.

[0121] Specifically, first, the final optimization model is used to perform text character recognition. The trained final optimization model is deployed to an actual text character recognition scenario (such as a document scanning OCR system, a PDF to structured text tool, etc.), and is used to perform recognition tasks on a to-be-recognized document image (which may be a real scenario document uploaded by a user, such as an office contract, a scientific research paper, a course lecture note, etc., and usually contains structured elements such as formulas, tables, and complex layouts). The model converts the text content in the document image into a structured string containing format markers (such as LaTeX format formulas, HTML table markers, etc.) according to a preset structured output logic, and outputs a recognition result.

[0122] The output result of the final optimization model has format specification, character accuracy, and syntax compliance given by the two-stage fine-tuning training (SFT+reinforcement learning) in the early stage, but there may be a small amount of recognition bias that is not covered by the training data due to the individualized characteristics of the document in the actual scenario (such as handwritten annotation interference, special fonts, non-standard layouts, etc.).

[0123] Then, the user's correction data for the recognition result can be obtained. For example, the system presents the recognition result output by the model to the user, allows the user to manually correct the recognition deviation, and finally collects the "correction data" containing the corrected standard answer, i.e., the complete structured text corrected by the user for the incorrect part in the recognition result (such as incorrect formula character writing, missing table row and column labels, list level confusion, etc.), which is the accurate labeling that meets the user's actual needs. Optionally, the correction data can also include error labeling information: the user can label the error type (such as "missing formula separator", "table label not closed", "character recognition error"), which provides more accurate direction for subsequent model targeted optimization. For example, if the model misrecognizes the formula "$F=ma$" in the document as "$F=ma$" (missing end separator "$$"), the user corrects and supplements the end separator to form the "to-be-recognized document image-corrected structured text" paired data, which is the correction data.

[0124] Subsequently, the second training image set is expanded based on the correction data. The collected "to-be-recognized document image-user correction data" paired data is preprocessed according to the standard of the previous second training image set (user scene sample image + structured labeling) (such as document image standardization, correction data tokenization, format label verification, etc.), and then added to the original second training image set to complete the incremental update of the data set. The core reason for updating the second training image set is that the original positioning of the second training image set is "sample image and corresponding structured labeling in user scenario", which is highly consistent with the scene attribute of the user feedback correction data (both come from real application scenarios); by continuously supplementing such data, the scene coverage of the second training image set can be continuously enriched (such as adding handwritten annotation documents, special industry format documents, etc.), which makes up for the scene gap of the initial training data, and provides more actual demand-oriented training samples for subsequent model optimization.

[0125] Finally, based on the updated second training image set, the existing "final optimization model" is incrementally trained. Incremental training is different from full training from scratch. Incremental training only fine-tunes the local parameters of the original model parameters using the newly added user feedback data, and the core goal is to enable the model to quickly learn the recognition rules of new scenarios and new cases while retaining the existing recognition capabilities, avoiding degradation of the original capabilities due to new data training. During training, the reinforcement learning fine-tuning logic of the previous second stage is followed (such as evaluating the model output based on a composite reward function, and updating the parameters through the PPO algorithm), and the learning rate can be appropriately reduced (such as using 1 / 10 of the initial learning rate) and the training rounds can be reduced, focusing on the error types corresponding to the correction data (such as "special font character recognition error" and "handwritten formula format deviation" that the user frequently corrects, to strengthen the model's learning of these features).

[0126] Through incremental training, the original "final optimization model" will be iterated into a new version with better adaptability, which not only solves the historical recognition bias, but also significantly improves the document recognition accuracy in future similar scenarios; and with the continuous accumulation of user feedback data and the continuous updating of the training set, the model will form a continuous iteration closed loop of "application-feedback-optimization", and long-term maintain high recognition performance in actual scenarios.

[0127] In summary, in the text recognition task scenario, by applying the technical solutions provided in the embodiments of the present application, a first training image set is obtained, and the pre-trained multi-modal large model is supervised fine-tuning trained using the first training image set, thereby obtaining a preliminary optimization model. Subsequently, a second training image set is further obtained, and a composite reward function is constructed, and based on the second training image set and the composite reward function, the preliminary optimization model is fine-tuned trained based on reinforcement learning, and finally an optimization model suitable for the text character recognition task is obtained. The scheme proposes a two-stage differentiated fine-tuning model strategy, aiming to adapt to the task characteristics and model optimization needs in different training stages. In the first stage, the goal is to cultivate general structured recognition ability, relying on standardized paired data and supervised learning to lay the foundation for the model. The second stage focuses on the improvement of actual scene adaptability and output quality, combining user real scene samples and multi-dimensional composite reward function to achieve precise optimization, effectively realizing the layer-by-layer iteration and overall optimization of model performance. At the same time, the second stage collects sample images based on user actual application scenarios, ensuring that the model optimization direction is deeply consistent with real use requirements, significantly improving the generalization ability and practical value of the model in actual scenarios. In addition, through structured text labeling, the output text of the final optimization model is a structured string with format markers, which can be correctly parsed and visualized by downstream rendering engines, layout restoration tools and other processing systems, completely solving the format loss and downstream processing failure problem caused by traditional OCR outputting only pure text.

[0128] Of course, in actual application, the technical solutions provided in the embodiments of the present application are not limited to the text recognition task scenario, and the core two-stage fine-tuning framework and composite reward optimization mechanism have high universality and can be widely applied to various artificial intelligence tasks that need to generate high-precision, structured output from unstructured or semi-structured input. For example, in the visual content description and visual question answering scenario, the model needs to generate accurate and fluent natural language descriptions or answers to related questions based on input images.

[0129] In this scenario, the application process of the technical solutions provided in the embodiments of the present application generally includes: First stage: supervised fine-tuning using a dataset containing images and corresponding detailed descriptions (such as structured information containing object attributes, spatial relationships, actions, etc.), so that the model learns basic description ability.

[0130] The second stage: a composite reward function is constructed to evaluate the accuracy of the generated description (the matching degree with the true value of the key entity), fluency (language model score), richness (the diversity of description details), etc. Exemplarily, in this scenario, the composite reward function is composed of at least two of the following loss item calculation functions, and the composite reward function is obtained by weighted summation of the weight coefficients of each loss item: a semantic matching loss item calculation function for evaluating the semantic consistency of the generated description with the true label on key entities, actions and relationships; a language fluency loss item calculation function for evaluating the naturalness and grammatical correctness of the generated description; and an information amount loss item calculation function for evaluating the detail richness and redundancy of the generated description.

[0131] Subsequently, the user's real scene image (such as an e-commerce product image or a medical image) is used to optimize the reinforcement learning in combination with the above-mentioned composite reward function, so that the generated description is more in line with the requirements of a specific field.

[0132] In summary, the two-stage fine-tuning and composite reward optimization method proposed in the present application has the core advantage of being able to guide the model to progress from learning "basic mapping ability" to mastering "high-quality output ability that meets the needs of complex scenarios". This paradigm can be transferred to any field involving conversion from raw data to structured information, and has wide application prospects.

[0133] Figure 5 An embodiment block diagram of a model training device according to an embodiment of the present application is provided. As shown in the figure, the device comprises: Figure 5 A first image set acquisition module 51 is configured to acquire a first training image set. A first stage training module 52 is configured to perform first fine-tuning training on a pre-trained artificial intelligence model using the first training image set to obtain a preliminary optimization model. A second image set acquisition module 53 is configured to acquire a second training image set and construct a composite reward function. The composite reward function is used to quantitatively evaluate the quality of the model output in multiple dimensions. A second stage training module 54 is configured to perform second fine-tuning training on the preliminary optimization model based on the second training image set and the composite reward function to obtain a final optimization model.

[0134] In a possible implementation, the final optimization model is used for a text character recognition task, the composite reward function is composed of at least two of the following loss item calculation functions, and the composite reward function is obtained by weighted summation of the weight coefficients of each loss item: a format loss item calculation function, a character accuracy loss item calculation function, and a grammatical correctness loss item calculation function. ​

[0135] In a possible implementation, the apparatus further includes: a format loss term calculation module configured to identify boundary information of structured elements in a predicted string output by the model and in a text-labeled string, respectively; compare the boundary information of the structured elements in the predicted string and the text-labeled string, identify format errors existing in the predicted string, and determine types of the format errors; assign a penalty value to each of the identified format errors according to the types of the format errors, and perform weighted summation on penalty values of all the format errors according to preset weights to obtain a format loss value.

[0136] In a possible implementation, the format loss term calculation module identifies boundary information of structured elements in a predicted string output by the model and in a text-labeled string, respectively, including: perform scanning on the predicted string output by the model and the corresponding text-labeled string based on a predefined format rule set respectively by using a regular expression matcher, and extract boundary information of each structured element in the predicted string and the text-labeled string; wherein the format rule set includes boundary identification rules for the structured elements; and the boundary information includes a starting position coordinate and an ending position coordinate of the structured element in the corresponding string.

[0137] In a possible implementation, the apparatus further includes: a character accuracy loss term calculation module configured to convert the predicted string output by the model and the text-labeled string into sequence representations composed of multiple processing units, respectively; determine a minimum edit distance between the sequence representations of the predicted string and the text-labeled string, respectively; the minimum edit distance is defined as a minimum number of single-character edit operations required to convert the predicted string into the text-labeled string; map the minimum edit distance into a character accuracy loss value; wherein the character accuracy loss value is positively correlated with the minimum edit distance.

[0138] In a possible implementation, the apparatus further includes: a syntax correctness loss term calculation module configured to extract structured elements in the predicted string output by the model; perform syntax compliance analysis on the structured elements by using a syntax parser; identify syntax errors existing in the predicted string based on the syntax compliance analysis result, and determine types of the syntax errors; According to the type of the syntax error, a penalty value is assigned to each identified syntax error, and penalty values of all syntax errors are weighted and summed according to a preset weight to obtain a syntax correctness loss value.

[0139] In a possible implementation, the second stage training module 54 is specifically configured to: input a sample image in the second training image set into the preliminary optimization model to generate at least two candidate output results for the sample image; for the same sample image, call the composite reward function to calculate a total loss value of each candidate output result of the sample image pair, and determine a reward value corresponding to each candidate output result based on the total loss value; wherein the reward value is negatively correlated with the total loss value; identify the first type of candidate output result and the second type of candidate output result by comparing the relative difference of the reward values between the candidate output results, and construct a preference signal for model optimization based on the identification result; maximize the probability expectation of the preliminary optimization model generating the first type of candidate output result, update the parameters of the preliminary optimization model based on the preference signal through the back propagation algorithm; repeat the above steps for iterative training until a preset iteration stopping condition is met, and obtain the final optimization model.

[0140] In a possible implementation, the first training image set includes original document images and structured text annotations corresponding to the original document images; and the first stage training module 52 is specifically configured to: perform supervised fine-tuning training on the pre-trained artificial intelligence model by using the first training image set to obtain the preliminary optimization model.

[0141] In a possible implementation, the apparatus further includes: a model optimization module configured to, after obtaining the final optimization model, perform text character recognition on a to-be-recognized document image by using the final optimization model, and output a recognition result; obtain correction data of the recognition result from a user; update the second training image set according to the correction data and the to-be-recognized document image, and perform incremental training on the final optimization model by using the updated second training image set.

[0142] As shown in Figure 6 Embodiments of the present application provide an electronic device, which includes a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112 and the memory 113 complete communication with each other through the communication bus 114, The memory 113 is configured to store a computer program. In an embodiment of the present application, the processor 111 is configured to implement the model training method provided by any one of the preceding method embodiments when executing the program stored in the memory 113. obtain a first training image set; perform first fine-tuning training on the pre-trained artificial intelligence model by using the first training image set to obtain a preliminary optimization model; obtain a second training image set and construct a composite reward function, wherein the composite reward function is used for multi-dimension quantitative evaluation of the quality of the model output; perform second fine-tuning training on the preliminary optimization model based on the second training image set and the composite reward function to obtain a final optimization model.

[0143] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the model training method provided by any one of the preceding method embodiments.

[0144] The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., may be located in one place, or may be distributed on a plurality of network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0145] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions essentially or in other words make contributions to the related art, and can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0146] It is to be understood that the terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises", "comprising", "includes", "including" and "has" are inclusive and therefore specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring their performance in the particular order

[0147] The above description is that of current embodiments of the application. Various modifications and changes can be made thereto without departing from the spirit and scope of the application as set forth. The scope of the application is not to be limited to the exact details shown above. < / n> < / m> < / m> < / m>

Claims

1. A model training method, characterized in that, The method comprises: obtaining a first training image set; performing first fine-tuning training on a pre-trained artificial intelligence model using the first training image set to obtain a preliminary optimization model; obtaining a second training image set and constructing a composite reward function; wherein the composite reward function is used for multi-dimensional quantitative evaluation of the quality of the model output; based on the second training image set and the composite reward function, performing second fine-tuning training on the preliminary optimization model to obtain a final optimization model.

2. The method of claim 1, wherein, The final optimization model is used for text character recognition tasks, the composite reward function is composed of at least two of the following loss term calculation functions, and the composite reward function is obtained by weighting and summing the weight coefficients of each loss term: a format loss term calculation function, a character accuracy loss term calculation function, and a syntax correctness loss term calculation function.

3. The method of claim 2, wherein, The calculation process of the format loss term calculation function comprises: respectively identifying the boundary information of the structured elements in the predicted string and the text annotation string output by the model; comparing the boundary information of the structured elements in the predicted string and the text annotation string, identifying the format errors existing in the predicted string, and determining the types of the format errors; according to the types of the format errors, assigning a penalty value to each identified format error, and weighting and summing the penalty values of all format errors according to a preset weight to obtain a format loss value.

4. The method of claim 3, wherein, The process of respectively identifying the boundary information of the structured elements in the predicted string and the text annotation string output by the model comprises: using a regular expression matcher to scan the predicted string and the corresponding text annotation string output by the model based on a pre-defined format rule set to extract the boundary information of each structured element in the predicted string and the text annotation string; wherein the format rule set contains boundary identification rules for structured elements; the boundary information includes the starting position coordinates and the ending position coordinates of the structured elements in the corresponding string.

5. The method of claim 2, wherein, The calculation process of the character accuracy loss term calculation function comprises: converting the predicted string and the text annotation string output by the model into sequence representations composed of multiple processing units, respectively; determining the minimum edit distance between the sequence representations of the predicted string and the text annotation string, respectively; the minimum edit distance is defined as the minimum number of single character editing operations required to convert the predicted string into the text annotation string; mapping the minimum edit distance to a character accuracy loss value; wherein the character accuracy loss value is positively correlated with the minimum edit distance.

6. The method of claim 2, wherein, The calculation process of the syntax correctness loss term calculation function comprises: extracting structured elements in the predicted string output by the model; using a syntax parser to perform syntax compliance analysis on the structured elements; based on the syntax compliance analysis result, identifying the syntax errors existing in the predicted string, and determining the types of the syntax errors; according to the types of the syntax errors, assigning a penalty value to each identified syntax error, and weighting and summing the penalty values of all syntax errors according to a preset weight to obtain a syntax correctness loss value.

7. The method according to any one of claims 1 to 6, characterized in that, The second fine-tuning training is performed on the preliminary optimization model based on the second training image set and the composite reward function to obtain a final optimization model, including: inputting a sample image in the second training image set into the preliminary optimization model to generate at least two candidate output results for the sample image; for the same sample image, calling the composite reward function to calculate a total loss value of each candidate output result of the sample image pair, and determining a reward value corresponding to each candidate output result based on the total loss value; wherein the reward value is negatively correlated with the total loss value; using a generalized penalty optimization algorithm to compare the relative differences in reward values between the candidate output results, identifying a first type of candidate output result and a second type of candidate output result, and constructing a preference signal for model optimization based on the identification result; maximizing the probability expectation of the preliminary optimization model generating the first type of candidate output result as a target, updating parameters of the preliminary optimization model based on the preference signal through a back propagation algorithm; iteratively training the above steps until a preset iteration stopping condition is met to obtain the final optimization model.

8. The method of claim 1, wherein, The first training image set includes original document images and structured text annotations corresponding to the original document images; and the first fine-tuning training is performed on the pre-trained artificial intelligence model using the first training image set to obtain a preliminary optimization model, including: performing supervised fine-tuning training on the pre-trained artificial intelligence model using the first training image set to obtain the preliminary optimization model.

9. The method of claim 1, wherein, After obtaining the final optimization model, the method further includes: performing text character recognition on a to-be-recognized document image using the final optimization model to output a recognition result; obtaining correction data of the user on the recognition result; updating the second training image set according to the correction data and the to-be-recognized document image, and performing incremental training on the final optimization model using the updated second training image set.

10. A model training apparatus, comprising: The device includes: a first image set acquisition module configured to acquire a first training image set; a first stage training module configured to perform first fine-tuning training on a pre-trained artificial intelligence model using the first training image set to obtain a preliminary optimization model; a second image set acquisition module configured to acquire a second training image set and construct a composite reward function; wherein the composite reward function is used for multi-dimensionally quantitatively evaluating the quality of model output; a second stage training module configured to perform second fine-tuning training on the preliminary optimization model based on the second training image set and the composite reward function to obtain a final optimization model.

Images