A handwritten mathematical expression recognition model training method, recognition method and device

Abstract

The application provides a handwritten mathematical expression recognition model training method, a recognition method and a device. LaTeX sequences are automatically extracted and enhanced from academic PDF files, and then a large-scale and high-real handwritten formula image dataset is generated through rendering and material conversion, and then an end-to-end recognition model is trained. The scheme fundamentally solves the core bottleneck of training data scarcity and diversity deficiency caused by high artificial annotation cost; through the synthesis of realistic handwritten images, the generalization ability and robustness of the model in the real scene are effectively improved; finally, the recognition model realizes a significant breakthrough in performance when processing complex and long sequence formulas, and provides a complete and scalable engineering solution for the large-scale application of the technology.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a training method, recognition method and apparatus for a handwritten mathematical expression recognition model. Background Technology

[0002] Handwritten Mathematical Expression Recognition (HMER) is an important branch of optical character recognition, aiming to automatically convert images of handwritten mathematical formulas into structured markup languages ​​(such as LaTeX). This technology has broad application prospects in fields such as digital education and scientific document archiving.

[0003] Currently, the development of this technology is mainly limited by the scarcity of training data. Due to the complex structure of mathematical expressions and the need for specialized knowledge in annotation, the cost of constructing large-scale, high-quality handwritten formula datasets is extremely high, resulting in a limited scale of existing public datasets (such as the CROHME series and HME100K). To address the lack of data, existing technical solutions mainly focus on two aspects: First, introducing strong inductive biases into the model structure, such as designing coverage attention, tree-like decoders, or joint symbol counting tasks, to leverage prior knowledge to guide learning. However, this makes the model structure complex and its ability to handle long-distance, complex grammatical expressions is poor. Second, performing synthetic augmentation at the data level, such as generating handwritten samples or combining isolated symbols through style transfer. However, these methods generate samples with limited diversity and have bottlenecks in terms of handwriting realism and expression complexity, making it difficult to achieve effective scalability. Furthermore, current mainstream evaluation metrics (such as ExpRate and BLEU) rely on precise string matching, failing to fully consider the two-dimensional structural characteristics of mathematical formulas and the rigor of LaTeX syntax. They are not sensitive enough to rendering failures caused by minor grammatical errors (such as missing brackets), resulting in evaluation results that are not entirely consistent with visual representation.

[0004] While the scaling law of data and model size has been proven to significantly improve performance in large-scale language models and general OCR, the scalability of model training using massive amounts of data remains largely unexplored and unpracticed in the field of Handwritten Mathematical Expression Recognition (HMER). Therefore, existing technologies suffer from data scarcity and low-quality data synthesis, leading to limited generalization ability of complex models and hindering the application of HMER technology in real-world complex scenarios. Thus, a new scheme for recognizing handwritten mathematical expressions is urgently needed. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a training method, recognition method and apparatus for a handwritten mathematical expression recognition model, so as to eliminate or improve one or more defects existing in the prior art and solve the model performance bottleneck problem caused by the scarcity of high-quality training data in the field of handwritten mathematical expression recognition.

[0006] On one hand, the present invention provides a method for training a handwritten mathematical expression recognition model, the method comprising the following steps:

[0007] PDF papers containing mathematical expressions are obtained based on one or more preset data sources. They are then converted to Markdown format using Nougat and LaTeX sequences are extracted. The LaTeX sequences are enhanced by expression concatenation, symbol replacement, and structural expansion to increase the data size.

[0008] The LaTeX sequence is rendered to form a handwritten mathematical expression image, and paper texture is added through a material conversion model to obtain a first sample image containing the handwritten mathematical expression.

[0009] Construct a first training sample set containing a preset number of samples, where each sample contains an image of the first sample and adds a corresponding LaTeX sequence label;

[0010] The initial formula recognition model is trained using the first training sample set. The first sample image is used as input and the prediction result of the LaTeX sequence is output. A loss function is established based on the deviation between the prediction result and the LaTeX sequence label to update the parameters of the initial formula recognition model, thereby obtaining a handwritten mathematical expression recognition model.

[0011] In some embodiments, the method further includes, after converting to Markdown format using Nougat and extracting the LaTeX sequence:

[0012] The extracted LaTeX sequences are deduplicated by calculating edit distance;

[0013] The LaTeX sequence is validated and rendered to obtain a validation image. Validation images with dimensions exceeding a set threshold are removed. A pre-trained formula detection model is used to check the validation images, and validation images that do not contain formulas or whose formulas are not centered are removed. The criterion for determining that a formula is not centered is that the center position of the formula in the validation image deviates from the center position of the image by a set proportion of the length or width of the validation image.

[0014] In some embodiments, the pre-training step of the formula detection model includes:

[0015] Obtain a second training sample set containing positive and negative samples. The positive samples are second sample images containing formulas and are labeled with positive sample identifiers, center positions, and edge boxes. The negative samples are second sample images that do not contain formulas and are labeled with all zeros.

[0016] The YOLOv8n model is trained using the second training sample set. The second sample image is input and the prediction results of whether the formula is contained and the center position and edge box are output. A classification loss is constructed based on the deviation between the prediction results of whether the second sample image contains the formula and the label. An offset loss is constructed based on the deviation between the prediction results of the center position and edge box of the formula in the positive sample and the label. The YOLOv8n model is updated by minimizing the classification loss and the offset loss to obtain the formula detection model.

[0017] In some embodiments, the LaTeX sequence is enhanced through expression concatenation, symbol substitution, and structural expansion, including:

[0018] Connect two or more formulas with operators, or construct them into matrices, multi-line expressions, or nest a single formula within other formulas;

[0019] Replace operators, variable identifiers, function names, and parenthesis types in the formula with similar types;

[0020] Add modifiers to variables in formulas to expand them with subscripts and superscripts, and expand them with fraction wrappers, square root wrappers, and function parameters; and expand the content by adding Chinese text and foreign letters.

[0021] In some embodiments, rendering the LaTeX sequence to form a handwritten mathematical expression image includes: rendering various handwritten mathematical expressions by loading a handwritten macro package, and adding random drift to simulate handwriting jitter effects;

[0022] The material conversion model adopts the diffusion-based Stable Diffusion, dream-textures, or material-diffusion, and introduces a cue word structure containing paper type, texture details, color, age, lighting, lens, and photography parameters to add paper texture to the handwritten mathematical expression image.

[0023] The initial formula recognition model adopts the TexTeller model, which includes a ViT image encoder and a Transformer text decoder. The ViT image encoder takes the first sample image as input and outputs an image feature vector sequence, and the Transformer text decoder outputs a LaTeX sequence.

[0024] In some embodiments, the method further includes:

[0025] Obtain a test set, wherein each test sample in the test set contains a handwritten mathematical expression test image and its corresponding standard LaTeX sequence;

[0026] The handwritten mathematical expression test image is input into the handwritten mathematical expression recognition model to obtain the detected LaTeX sequence;

[0027] The standard LaTeX sequence and the detected LaTeX sequence are rendered into images, and their pixels are aligned by dilation and translation operations. The proportion of the pixel overlap region to the joint region is calculated to obtain the extended pixel matching ratio, so as to evaluate the recognition effect of the handwritten mathematical expression recognition model and generate a detection report.

[0028] On the other hand, the present invention also provides a method for recognizing handwritten mathematical expressions, the method comprising:

[0029] The image containing the handwritten mathematical expression to be recognized is obtained and input into the handwritten mathematical expression recognition model obtained by the above-mentioned handwritten mathematical expression recognition model training method, and the LaTeX sequence of the handwritten mathematical expression is output.

[0030] On the other hand, the present invention also provides a handwritten mathematical expression recognition device, including a processor, a memory, and a computer program or instructions stored in the memory, wherein the processor is used to execute the computer program or instructions, and when the computer program or instructions are executed, the device implements the steps of the above method.

[0031] On the other hand, the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0032] On the other hand, the present invention also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0033] The handwritten mathematical expression recognition model training method, recognition method, and apparatus described in this invention automatically extracts and enhances LaTeX sequences from academic PDF files, then generates a large-scale, highly realistic dataset of handwritten formula images through rendering and material conversion, thereby training an end-to-end recognition model. This solution fundamentally solves the core bottleneck of scarce and insufficiently diverse training data caused by the high cost of manual annotation; by synthesizing realistic handwritten images, it effectively improves the model's generalization ability and robustness in real-world scenarios; ultimately, it enables the recognition model to achieve a significant performance breakthrough when processing complex, long-sequence formulas, and provides a complete and scalable engineering solution for the large-scale application of this technology.

[0034] Furthermore, the LaTeX sequences extracted from academic PDF files were deduplicated by calculating edit distance, and the images that could not effectively form handwritten mathematical expressions or had quality problems were removed by verification rendering, thus improving the data quality.

[0035] Furthermore, the extracted LaTeX sequences are automatically enhanced through a three-level operation of expression concatenation, symbol replacement, and structural expansion, thereby expanding the data scale.

[0036] Furthermore, by loading a handwriting macro package and introducing random jitter, realistic handwriting is simulated; then, a material conversion technique based on a diffusion model is adopted, combined with refined prompts describing the physical properties of paper and imaging conditions, to synthesize highly realistic paper backgrounds and material effects for handwriting images, thereby improving the model's generalization ability and robustness in practical applications.

[0037] Furthermore, the Extended Pixel Match Ratio (EPMR) is adopted as the core evaluation metric, which overcomes the shortcomings of traditional string matching metrics that are too strict on syntax and insensitive to visual consistency. It provides a reliable evaluation standard that is more in line with human visual judgment and can better reflect the actual application performance of the model, thus ensuring the effectiveness of the training and evaluation process.

[0038] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the text, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0039] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0040] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0041] Figure 1 This is a flowchart of a handwritten mathematical expression recognition model training method according to an embodiment of the present invention.

[0042] Figure 2 This is a logical schematic diagram of the handwritten mathematical expression recognition model training method according to another embodiment of the present invention.

[0043] Figure 3 This is an image showing the effect of adding paper texture through a material conversion model in the handwritten mathematical expression recognition model training method according to another embodiment of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0045] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0046] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0047] Current handwritten mathematical expression recognition technologies face three major bottlenecks: First, training data is severely scarce, relying on limited handwritten datasets and synthesis methods, resulting in insufficient diversity, poor realism, and difficulty in covering complex and long formulas. Second, model design is complex and has limited generalization ability; strong inductive bias or complex decoding mechanisms are introduced to compensate for insufficient data, leading to poor recognition performance for complex structures and long sequence formulas. Third, the evaluation system has flaws; widely used string matching metrics cannot effectively evaluate the two-dimensional visual layout and grammatical correctness of formulas, causing discrepancies between evaluation results and actual recognition performance. These fundamental problems collectively restrict the development of this technology towards high performance and practical application.

[0048] In view of this, the present invention provides a method for training a handwritten mathematical expression recognition model, such as... Figure 1 As shown and Figure 2 As shown, the method includes the following steps S101~S104:

[0049] Step S101: Obtain PDF format papers containing mathematical expressions based on one or more preset data sources, convert them to Markdown format using Nougat and extract LaTeX sequences, and enhance the LaTeX sequences by expression concatenation, symbol replacement and structural expansion to increase the data scale.

[0050] Step S102: Render the LaTeX sequence to form a handwritten mathematical expression image, and process it using a material conversion model to add paper texture to obtain the first sample image containing the handwritten mathematical expression.

[0051] Step S103: Construct a first training sample set containing a preset number of samples, each sample containing a first sample image and adding a corresponding LaTeX sequence label.

[0052] Step S104: Train the initial formula recognition model using the first training sample set, take the first sample image as input and output the prediction result of the LaTeX sequence, establish a loss function based on the deviation between the prediction result and the LaTeX sequence label to update the parameters of the initial formula recognition model, and obtain the handwritten mathematical expression recognition model.

[0053] Step S101 is used for the extraction and enhancement of LaTeX sequences, aiming to build a large-scale, high-quality corpus of mathematical expressions. In practice, PDF papers are first obtained from publicly available academic platforms such as arXiv and parsed using the Nougat model. This model can accurately convert PDF layouts containing complex macro definitions into Markdown format and extract clean LaTeX sequences, effectively solving the rendering compatibility issues caused by custom macro commands in the original source code.

[0054] In some embodiments, LaTeX sequences are enhanced through expression concatenation, symbol substitution, and structural expansion, including: connecting two or more formulas with operators, constructing them into matrices or multi-line expressions, or nesting a single formula within other formulas; performing similar substitutions for operators, variable identifiers, function names, and bracket types in formulas; adding modifiers to variables in formulas to expand their subscripts and superscripts and their limits, and expanding them by adding fraction wrappers, square root wrappers, and function parameters; and expanding the content by adding Chinese text and foreign language letters.

[0055] Specifically, the extracted sequences undergo a three-level enhancement process, including: 1) Expression concatenation, such as connecting "a+b" and "\frac{c}{d}" with an equals sign to form "a+b = \frac{c}{d}", or vertically stacking equations to construct a system. 2) Symbol substitution, systematically converting operators, variables, or bracket types to synonyms, such as replacing "\times" with "\cdot" and "x" with "\xi". 3) Structural expansion, adding subscripts and superscripts to variables, such as expanding "x" to "x_{i}^{2}", or wrapping expressions as radicals or fractions. The technical effect is to automatically generate a massive amount of new expressions with high structural and symbolic diversity from limited seed data, completely solving the fundamental bottleneck of scarce and insufficient training data, and laying a data foundation for the model to learn complex mathematical grammar.

[0056] In some embodiments, the method further includes steps S1011 and S1012 after converting Nougat to Markdown format and extracting the LaTeX sequence:

[0057] Step S1011: Deduplication of the extracted LaTeX sequences is performed by calculating edit distance. The large number of LaTeX sequences extracted and enhanced in Step S101 are refined for deduplication to improve the diversity and information density of the dataset. Specifically, edit distance (Levenshtein distance) is used as the quantitative indicator of sequence similarity. Edit distance refers to the minimum number of single-character edits required to convert one string into another. The system calculates the edit distance pairwise for each batch of LaTeX sequences to be processed. When the edit distance between two sequences is lower than a preset threshold, they are considered approximately duplicates. For example, the sequence "\alpha + \beta" has a distance of 0 with "\alpha + \beta", indicating a complete duplicate; and a distance of 1 with "\alpha + \gamma", indicating a need for one replacement. At a threshold of 1, both are considered duplicates and are removed. This effectively filters out redundant samples extracted from the same paper or generated through enhancement that differ only in individual characters or spaces, ensuring that each sample in the dataset carries as much new information as possible. To prevent the model from memorizing highly similar but non-critical minor patterns, the model is forced to learn more fundamental mathematical structures and grammatical rules, thereby making more robust inferences when encountering new samples and reducing the probability of generating meaningless or repetitive fragments.

[0058] Step S1012: Perform verification rendering on the LaTeX sequence to obtain verification images, and remove verification images whose length and width exceed the set threshold; use a pre-trained formula detection model to check the verification images, and remove verification images that do not contain formulas or whose formula positions are not centered; wherein, the criterion for determining that the position is not centered is: the center position of the formula in the verification image deviates from the center position of the image by more than the set proportion of the length or width of the verification image.

[0059] Step S1012, based on sequence deduplication, further performs rigorous image-level quality filtering to ensure that each image-label pair used for training meets preset visual and content standards. The specific implementation consists of three sub-steps: 1) Rendering verification and size filtering: First, all LaTeX sequences are attempted to be rendered as images. For successfully rendered images, their aspect ratio or absolute size is calculated, and abnormal images exceeding a set threshold in length or width are removed. This aims to exclude images with out-of-control sizes due to LaTeX code errors or rendering engine malfunctions. 2) Formula content and integrity verification: A pre-trained formula detection model is used to analyze the rendered images. This model determines whether the image contains a valid mathematical formula region. If the detection model determines that the image does not contain any formulas, the entire sample is removed to ensure the validity of the training data content. 3) Formula position centering verification: For images where formulas are detected, the coordinates of the center point of the bounding rectangle of the formula region are further calculated and compared with the coordinates of the center point of the image. If the center of the formula deviates from the image center in the horizontal or vertical direction by more than a set proportion of the image width or height (e.g., more than 10%), it is judged as "not centered" and discarded. This operation enforces the normalization of the training data in terms of composition.

[0060] Furthermore, in some embodiments, the pre-training step of the formula detection model includes steps S201-S202:

[0061] Step S201: Obtain a second training sample set containing positive and negative samples. Positive samples are second sample images containing formulas and are labeled with positive sample identifiers, center positions, and edge boxes. Negative samples are second sample images that do not contain formulas and are labeled with all-zero labels.

[0062] Step S202: Train the YOLOv8n model using the second training sample set. Input the second sample image and output the prediction results of whether the formula is contained and the center position and edge box. Construct a classification loss based on the deviation between the prediction results of whether the second sample image contains the formula and the label. Construct an offset loss based on the deviation between the prediction results of the center position and edge box of the formula in the positive sample and the label. Minimize the classification loss and offset loss to update the parameters of the YOLOv8n model to obtain the formula detection model.

[0063] Specifically, in step S201, a second training sample set specifically for object detection is constructed. Positive samples are images containing clear mathematical formulas, labeled with their category identifier, the coordinates of the center point of the formula region, and the coordinates of its bounding rectangle. Negative samples are interference images that do not contain any formulas (such as plain text paragraphs or blank backgrounds), labeled with all-zero vectors to distinguish them. In step S202, the lightweight and efficient YOLOv8n model architecture is used for training, simultaneously performing classification and regression tasks: the model takes the second sample image as input, outputting a prediction of whether the image contains a formula (classification task), calculating the deviation between the prediction and the true category label using cross-entropy loss to constitute the classification loss; on the other hand, for regions predicted to contain formulas, the model outputs their predicted center point and bounding box coordinates (regression task), calculating the deviation between the predicted center point and the true location label using mean squared error or CIoU loss to constitute the offset loss. The training process obtains the total loss by weighted summation of classification loss and offset loss, and backpropagates to minimize this total loss, thereby iteratively updating the parameters of the YOLOv8n model. Finally, a dedicated detection model is obtained that can quickly and accurately locate and select mathematical formula regions in complex images, providing a reliable core component for the automation process in the aforementioned data cleaning workflow.

[0064] Step S102 is used for rendering and realizing the handwritten image, responsible for converting the enhanced LaTeX sequence into a realistic handwritten image. The implementation is divided into two stages: First, handwriting rendering, which uses a specific handwriting macro package (such as `handwriting`) from the LaTeX engine to render the sequence as digital handwriting, introducing random positional jitter and stroke thickness variations to simulate the non-rigid characteristics of real writing; second, material synthesis, where the rendered handwriting image is used as the foreground and input into a pre-trained diffusion model, such as `Stable Diffusion`. By constructing detailed text prompts, such as "a slightly yellowed old paper with a fibrous texture, illuminated by natural sidelight," the model is driven to generate a background that coordinates with the image and has a highly realistic paper texture. Finally, the handwriting and background are synthesized using image fusion technology. This overcomes the limitations of traditional synthesized images, which often have a "stiff" texture and a monotonous background, generating image samples that visually closely approximate real handwriting scenes. This greatly enhances the realism and immersion of the training data, and is key to the model's powerful real-world generalization ability.

[0065] In some embodiments, rendering a LaTeX sequence to form a handwritten mathematical expression image includes: rendering various handwritten mathematical expressions by loading a handwritten macro package, and adding random drift to simulate handwriting jitter effects.

[0066] Specifically, the LaTeX rendering engine calls specialized handwriting macro packages (such as handwriting.sty or the boondox font family) to convert standard mathematical symbols into handwritten glyphs with continuous strokes and natural cursive writing. To achieve more realistic dynamic effects, algorithmic random perturbations are introduced into the rendering coordinates. When generating each character or stroke, a small random translation (±1~3 pixels) and rotation are applied to its reference coordinates, supplemented by random fluctuations in stroke thickness. For example, when rendering the expression "x+y", the center of the character "x" may shift to the upper right corner relative to its theoretical position, and a slight ink buildup simulation appears at the stroke junctions. The two strokes of the plus sign "+" may appear not completely perpendicular and have uneven thickness. This breaks the absolutely regular mechanical feel of printed text, injects the non-rigid characteristics inherent in natural handwriting, and provides the model with key visual cues for learning the fluctuations in real handwriting.

[0067] The material conversion model employs diffusion-based methods such as Stable Diffusion, dream-textures, or material-diffusion. It introduces cue word structures containing information about paper type, texture details, color, age, lighting, lens type, and photographic parameters to add paper textures to handwritten mathematical expression images, achieving an effect similar to... Figure 3 As shown. For example, to simulate classroom notes, the prompt could be constructed as: "A slightly yellowed notebook paper with light gray horizontal lines, the paper has slight curling and wrinkles, and under soft indoor lighting, there is a faint reflection of pencil writing, with a blurred background." The diffusion model generates a highly realistic pure background texture image based on this prompt. Subsequently, digital image processing techniques (such as Multiply blending mode or Poisson fusion) are used to seamlessly blend the handwritten handwriting generated in the first stage (usually processed as a black or dark foreground) with the generated paper background, making the handwriting appear to "grow" on the paper texture and presenting a tone and shadow consistent with the ambient light. The technical effect is that it completely solves the core problems of fake and monotonous texture in synthetic images. Through precise control of the prompt, an infinite variety of material backgrounds that conform to different real-world scenarios can be generated as needed, making the synthetic data realistic enough to deceive human vision in terms of details such as texture, lighting, and wear, thereby greatly improving the generalization ability and robustness of the recognition model trained on this data in real-world complex scenarios.

[0068] Step S103 is used to construct the training sample set, which is the formatting process of data into the training carrier. Specifically, the first sample images generated in batches in step S102 are paired one-to-one with their corresponding LaTeX sequences, which serve as enhancement sources, to form a structured set of image-text label pairs. Based on this, a small number of real handwritten formula images and their annotations need to be mixed in according to a preset ratio to maintain the model's calibration against the distribution of real handwriting. This constructs a standardized training dataset that is controllable in scale, uniform in format, and contains real data anchors, providing direct input for subsequent repeatable and scalable model training.

[0069] Step S104 is used to train the recognition model, completing the training of the end-to-end recognition model. Specifically, the initial formula recognition model uses the TeXTeller model, which includes an OViT image encoder and a Transformer text decoder. The OViT image encoder takes the first sample image as input and outputs a sequence of image feature vectors, while the Transformer text decoder outputs a LaTeX sequence. During training, the first sample image is used as input, and its corresponding LaTeX label is used as the training target. The cross-entropy loss function is used to calculate the deviation between the model's predicted sequence and the target sequence, and the model parameters are iteratively optimized through backpropagation. The technical effect is that by utilizing the large-scale, highly realistic data generated in the aforementioned steps, a powerful general-purpose architecture is driven to learn, enabling the model to directly capture complex two-dimensional mathematical structural relationships from pixel-level images and map them into accurate grammatical sequences. Ultimately, a practical model is trained that exhibits excellent recognition performance in complex formulas, long-distance dependencies, and diverse writing styles.

[0070] In some embodiments, the method further includes steps S301 to S303:

[0071] Step S301: Obtain the test set. Each test sample in the test set contains a handwritten mathematical expression test image and its corresponding standard LaTeX sequence.

[0072] Step S302: Input the handwritten mathematical expression test image into the handwritten mathematical expression recognition model to obtain the detected LaTeX sequence.

[0073] Step S303: Render the standard LaTeX sequence and the detected LaTeX sequence into an image, and align the pixels through dilation and translation operations. Calculate the proportion of the pixel overlap region to the joint region to obtain the extended pixel matching ratio, in order to evaluate the recognition effect of the handwritten mathematical expression recognition model and generate a detection report.

[0074] This embodiment provides a model evaluation method that constructs a closed-loop, visual consistency-based quantitative evaluation system. Specifically, step S301 first constructs a standard test set, where each sample consists of a real or highly realistic handwritten formula image and its accurate standard LaTeX sequence, providing an objective benchmark for evaluation. Step S302 executes model inference by inputting the test image into the trained handwritten mathematical expression recognition model to obtain its predicted detection LaTeX sequence, thereby acquiring the model's original output. The core evaluation occurs in step S303, using Extended Pixel Match Ratio (EPMR) as the evaluation metric. The standard sequence and the detected sequence are rendered into binary images using the same LaTeX engine. Then, pixel dilation is performed on the detected image to tolerate differences in handwriting thickness, and a systematic translation is performed within a small neighborhood defined by the dilSize parameter. The visual similarity between the two is quantified by calculating the proportion of the maximized pixel overlap area between the dilated and translated detected image and the standard image to the joint pixel area of ​​the two. It fundamentally overcomes the inherent shortcomings of traditional string matching metrics (such as Exprate and BLEU) that are too strict on LaTeX syntax and cannot effectively assess visual consistency. By simulating the error-tolerant mechanism of human visual judgment, EPMR can more reasonably and robustly reflect the real performance of the model in practical applications. In particular, it has a higher tolerance for non-critical syntax deviations (such as fine-tuning of bracket size and spacing) that occur in long formulas and complex structures. This provides more accurate feedback for model optimization than simple string comparison and generates a comprehensive quantitative test report that reflects the strengths and weaknesses of the model.

[0075] On the other hand, the present invention also provides a method for recognizing handwritten mathematical expressions, the method comprising:

[0076] The image containing the handwritten mathematical expression to be recognized is obtained and input into the handwritten mathematical expression recognition model obtained by the above-mentioned handwritten mathematical expression recognition model training method, and the LaTeX sequence of the handwritten mathematical expression is output.

[0077] On the other hand, the present invention also provides a handwritten mathematical expression recognition device, including a processor, a memory, and a computer program or instructions stored in the memory, wherein the processor is used to execute the computer program or instructions, and when the computer program or instructions are executed, the device implements the steps of the above method.

[0078] On the other hand, the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0079] On the other hand, the present invention also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0080] The present invention will now be described with reference to a specific embodiment:

[0081] This embodiment provides a scalable training method for handwritten mathematical expression recognition, such as... Figure 2 As shown, the process includes the following steps S1 to S6:

[0082] Step S1: Data Acquisition and Preprocessing. Mathematical paper PDFs are collected from platforms such as arXiv. LaTeX sequences are extracted using Nougat conversion, and macro characters are filtered to ensure syntactic consistency. This method has several significant advantages. First, the Markdown files output by the model no longer contain custom macro characters. Second, all LaTeX formulas originate from the same author, ensuring a high degree of consistency in the extracted expressions. This consistency management helps improve the efficiency and effectiveness of the entire data flow.

[0083] Step S2: Data augmentation and synthesis, which involves splicing formulas, replacing symbols, diversifying fonts, and enhancing the paper texture and ink marks generated by the diffusion model to improve the authenticity of the data.

[0084] Specifically, a diffusion model is used to generate realistic paper textures, which are then combined with LaTeX rendering tables featuring different fonts and inked to simulate effects such as ink bleeding or fading. This results in more diverse and accurate representations of the generated formula images.

[0085] Step S3: The TeX80M dataset was constructed, and the processed data was integrated to form 80 million high-quality samples, which were then released after deduplication and quality verification. Approximately 20% of the short formulas (within a set length) were retained in the dataset, including mathematical symbols and their combinations, to ensure the model did not degrade. About 3,500 commonly used Chinese words were randomly selected and added to the dataset after being mixed with the formulas. Although these Chinese texts have no semantic meaning, they enable the model to directly acquire OCR capabilities for Chinese scenarios.

[0086] The statistics of the Tex80M dataset are shown in Table 1 below.

[0087] Table 1

[0088]

[0089] Step S4: Model training, using the ViT+Transformer architecture of TeXTeller, mixed training of TeX80M and a small amount of handwritten data, and optimization of loss function and learning rate strategy.

[0090] Step S5: Evaluation and Validation. The model is evaluated using the TeX80M-Test benchmark and the Extended Pixel Match Ratio (EPMR) metric to validate its robustness in complex scenes. EPMR is a more intuitive and efficient metric that imposes stricter penalties on LaTeX syntax errors in the test samples. First, both the test and reference LaTeX images are converted to images. EPMR addresses pixel misalignment and achieves optimal alignment by dilating the pen pixels of the test image and translating it according to the `dilSize` parameter. The final EPMR score is the ratio of the overlapping area between the dilated test and reference images to their joint region, where the dilation operation smooths pixel shifts, thus enhancing robustness.

[0091] Step S6: Application Deployment. Apply the model to education systems, digitization of scientific research documents, and multimodal OCR scenarios to realize the practical application of the technology.

[0092] Meanwhile, a system implementation method is provided, including: a data acquisition module, an enhancement synthesis module, a hybrid training module, an evaluation module, and an application interface.

[0093] The data acquisition module automatically extracts PDF data from academic papers and extracts LaTeX sequences. The enhancement and synthesis module automates formula concatenation, symbol replacement, and font and texture enhancement. The hybrid training module supports multi-source data fusion and distributed training to optimize model parameters. The evaluation module integrates the EPMR metric and the TeX80M-Test benchmark, automatically generating performance reports. The application interface provides APIs and SDKs for easy integration into educational, research, and OCR products.

[0094] Table 2 shows the results in widely accepted HME benchmarks. Despite its simple design, TexTeller performs exceptionally well across all benchmarks, significantly outperforming other models.

[0095] Table 2

[0096]

[0097] Given the rapid development of large vision language models, their performance on the HMER task was explored. As shown in Table 3, the evaluation indicates that large vision language models are inferior to TexTeller in terms of accuracy and instruction compliance compared to the model presented in this embodiment. Furthermore, it was found that the Exprate metric has limited reliability in evaluating the performance of complex formulas.

[0098] Table 3

[0099]

[0100] Corresponding to the above method, the present invention also provides an apparatus / system including a computer device, the computer device including a processor and a memory, the memory storing computer instructions, the processor executing the computer instructions stored in the memory, and the steps of the apparatus / system method when the computer instructions are executed by the processor.

[0101] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. This computer-readable storage medium can be a tangible storage medium as described above.

[0102] Medium, such as random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0103] In summary, the handwritten mathematical expression recognition model training method, recognition method, and apparatus described in this invention automatically extracts and enhances LaTeX sequences from academic PDF files, then generates a large-scale, highly realistic dataset of handwritten formula images through rendering and material conversion, thereby training an end-to-end recognition model. This solution fundamentally solves the core bottleneck of scarce and insufficiently diverse training data caused by the high cost of manual annotation; by synthesizing realistic handwritten images, it effectively improves the model's generalization ability and robustness in real-world scenarios; ultimately, it enables the recognition model to achieve a significant performance breakthrough when processing complex, long-sequence formulas, and provides a complete and scalable engineering solution for the large-scale application of this technology.

[0104] Furthermore, the LaTeX sequences extracted from academic PDF files were deduplicated by calculating edit distance, and the images that could not effectively form handwritten mathematical expressions or had quality problems were removed by verification rendering, thus improving the data quality.

[0105] Furthermore, the extracted LaTeX sequences are automatically enhanced through a three-level operation of expression concatenation, symbol replacement, and structural expansion, thereby expanding the data scale.

[0106] Furthermore, by loading a handwriting macro package and introducing random jitter, realistic handwriting is simulated; then, a material conversion technique based on a diffusion model is adopted, combined with refined prompts describing the physical properties of paper and imaging conditions, to synthesize highly realistic paper backgrounds and material effects for handwriting images, thereby improving the model's generalization ability and robustness in practical applications.

[0107] Furthermore, the Extended Pixel Match Ratio (EPMR) is adopted as the core evaluation metric, which overcomes the shortcomings of traditional string matching metrics that are too strict on syntax and insensitive to visual consistency. It provides a reliable evaluation standard that is more in line with human visual judgment and can better reflect the actual application performance of the model, thus ensuring the effectiveness of the training and evaluation process.

[0108] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0109] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0110] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0111] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for training a handwritten mathematical expression recognition model, characterized in that, The method includes the following steps: PDF papers containing mathematical expressions are obtained based on one or more preset data sources. They are then converted to Markdown format using Nougat and LaTeX sequences are extracted. The LaTeX sequences are enhanced by expression concatenation, symbol replacement, and structural expansion to increase the data size. The LaTeX sequence is rendered to form a handwritten mathematical expression image, and paper texture is added through a material conversion model to obtain a first sample image containing the handwritten mathematical expression. Construct a first training sample set containing a preset number of samples, where each sample contains an image of the first sample and adds a corresponding LaTeX sequence label; The initial formula recognition model is trained using the first training sample set. The first sample image is used as input and the prediction result of the LaTeX sequence is output. A loss function is established based on the deviation between the prediction result and the LaTeX sequence label to update the parameters of the initial formula recognition model, thereby obtaining a handwritten mathematical expression recognition model. The process of rendering the LaTeX sequence to form a handwritten mathematical expression image includes: rendering various handwritten mathematical expressions by loading a handwritten macro package, and adding random drift to simulate handwriting jitter effects. The material conversion model adopts the diffusion-based Stable Diffusion, dream-textures, or material-diffusion, and introduces a cue word structure containing paper type, texture details, color, age, lighting, lens, and photography parameters to add paper texture to the handwritten mathematical expression image. The initial formula recognition model adopts the TexTeller model, which includes a ViT image encoder and a Transformer text decoder. The ViT image encoder takes the first sample image as input and outputs an image feature vector sequence, and the Transformer text decoder outputs a LaTeX sequence.

2. The handwritten mathematical expression recognition model training method according to claim 1, characterized in that, The method, after converting Nougat to Markdown format and extracting the LaTeX sequence, also includes: The extracted LaTeX sequences are deduplicated by calculating edit distance; The LaTeX sequence is validated and rendered to obtain a validation image. Validation images with dimensions exceeding a set threshold are removed. A pre-trained formula detection model is used to check the validation images, and validation images that do not contain formulas or whose formulas are not centered are removed. The criterion for determining that a formula is not centered is that the center position of the formula in the validation image deviates from the center position of the image by a set proportion of the length or width of the validation image.

3. The handwritten mathematical expression recognition model training method according to claim 2, characterized in that, The pre-training steps of the formula detection model include: Obtain a second training sample set containing positive and negative samples. The positive samples are second sample images containing formulas and are labeled with positive sample identifiers, center positions, and edge boxes. The negative samples are second sample images that do not contain formulas and are labeled with all zeros. The YOLOv8n model is trained using the second training sample set. The second sample image is input and the prediction results of whether the formula is contained and the center position and edge box are output. A classification loss is constructed based on the deviation between the prediction results of whether the second sample image contains the formula and the label. An offset loss is constructed based on the deviation between the prediction results of the center position and edge box of the formula in the positive sample and the label. The YOLOv8n model is updated by minimizing the classification loss and the offset loss to obtain the formula detection model.

4. The handwritten mathematical expression recognition model training method according to claim 1, characterized in that, The LaTeX sequence is enhanced through expression concatenation, symbol substitution, and structural expansion, including: Connect two or more formulas with operators, or construct them into matrices, multi-line expressions, or nest a single formula within other formulas; Replace operators, variable identifiers, function names, and parenthesis types in the formula with similar types; Add modifiers to variables in formulas to expand them with subscripts and superscripts, and expand them with fraction wrappers, square root wrappers, and function parameters; and expand the content by adding Chinese text and foreign letters.

5. The handwritten mathematical expression recognition model training method according to claim 1, characterized in that, The method further includes: Obtain a test set, wherein each test sample in the test set contains a handwritten mathematical expression test image and its corresponding standard LaTeX sequence; The handwritten mathematical expression test image is input into the handwritten mathematical expression recognition model to obtain the detected LaTeX sequence; The standard LaTeX sequence and the detected LaTeX sequence are rendered into images, and their pixels are aligned by dilation and translation operations. The proportion of the pixel overlap region to the joint region is calculated to obtain the extended pixel matching ratio, so as to evaluate the recognition effect of the handwritten mathematical expression recognition model and generate a detection report.

6. A method for recognizing handwritten mathematical expressions, characterized in that, The method includes: The image to be recognized, containing a handwritten mathematical expression, is obtained and input into the handwritten mathematical expression recognition model obtained by the handwritten mathematical expression recognition model training method according to any one of claims 1 to 5, and the LaTeX sequence of the handwritten mathematical expression is output.

7. A handwritten mathematical expression recognition device, comprising a processor, a memory, and a computer program or instructions stored in the memory, characterized in that, The processor is configured to execute the computer program or instructions, and when the computer program or instructions are executed, the device implements the steps of the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method described in any one of claims 1 to 5.

Images