An engineering scene formula recognition optimization method, device, equipment and storage medium

Abstract

The application discloses an engineering scene formula recognition optimization method and device, equipment and a storage medium. The method first constructs a Chinese formula data set based on a LaTeX seed formula data set; then a formula recognition base model is constructed and trained and optimized using the data set, a rendering feedback mechanism is introduced in the training and optimization process, the model prediction sequence is rendered into an image and compared with a target image, and sample weights are dynamically adjusted to focus on difficult samples; finally, in the inference recognition, the rendering feedback mechanism is also applied to render and score and reorder multiple candidate results generated by the model, and output the optimal recognition result. The application solves the problem of lack of Chinese formula data in the engineering field, and simultaneously improves the recognition accuracy of the model and the rendering reliability of the output result through the rendering feedback mechanism, and has good engineering practical value and model universality.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for formula recognition optimization in engineering scenarios. Background Technology

[0002] In engineering design research and technical exchange, engineering documents such as design specifications, technical manuals, engineering drawings, and research papers serve as carriers of core knowledge. Formulas, in particular, with their precision and conciseness, convey crucial technical information. With the advancement of information technology, the automatic and accurate identification and structured extraction of formulas from massive amounts of documents is a key step in achieving document content retrieval, building engineering knowledge bases, and supporting downstream applications such as automated calculations and simulations. Formulas within engineering documents possess the following characteristics:

[0003] (1) Complex structure: Formulas often contain multiple layers of parentheses, matrices, integral symbols, etc., and involve engineering-specific symbols such as mechanical vectors and thermodynamic parameters;

[0004] (2) Diverse styles: Engineering documents from different institutions and at different times have different styles of formula layout, fonts and symbols. Traditional methods based on templates or fixed rules have poor universality and are difficult to cope with diverse real-world scenarios.

[0005] (3) Image degradation: such as fading or blurring caused by old paper, watermark interference, scanning noise, etc.;

[0006] (4) Localization elements: In Chinese engineering practice, formulas often incorporate Chinese subscripts, Chinese variables, Chinese conjunctions, Chinese annotations, etc. These elements enhance the localization of the formulas, but increase the difficulty of recognition.

[0007] Currently, advanced methods for formula recognition in image-formatted documents employ deep learning-based visual encoder-decoder models to convert formula images into structured markup languages ​​such as LaTeX. This approach has proven effective for complex formula recognition and image degradation issues. However, in practical engineering applications, existing technologies still face the following challenges:

[0008] First, existing formula recognition models have low accuracy in recognizing formulas containing Chinese elements (such as Chinese variables, Chinese subscripts, and Chinese annotations). The fundamental reason for this is that such samples are extremely scarce in public datasets. Even when collecting data in real engineering scenarios, there are still problems such as incomplete coverage of Chinese characters and limited styles, which prevents the model from fully and effectively learning the mixed patterns of Chinese and mathematical symbols.

[0009] Secondly, traditional evaluation metrics cannot effectively measure the rendering reliability of LaTeX formulas. A LaTeX sequence with extremely high similarity to the standard answer text may fail to be correctly parsed by the rendering engine due to minor syntax errors, leading to rendering failure or display anomalies. Existing methods mostly use text similarity metrics such as edit distance or BLEU (Bilingual Evaluation Understudy) scores to assess recognition accuracy, but these metrics have low sensitivity to LaTeX syntax. For example, unclosed curly braces, the use of non-standard commands, or command syntax errors may only result in minor character differences, and the similarity index may still be high, but in actual rendering, it may cause problems such as rendering failure, symbol misalignment, or abnormal crashes. In engineering document scenarios, formula recognition accuracy and LaTeX rendering success rate directly affect the reliability and efficiency of downstream tasks. Rendering failures can lead to formula index errors in knowledge base construction, semantic deviations during knowledge extraction and retrieval, failures to import LaTeX into engineering design software such as CAD, interruptions in automated calculation processes, and the introduction of cascading errors, even posing security risks.

[0010] In summary, while existing formula recognition technologies are effective in basic scenarios, they are difficult to adapt to the localization needs of engineering documents and the stringent rendering requirements. Summary of the Invention

[0011] In view of this, the purpose of the present invention is to provide a formula recognition optimization method, apparatus, device and storage medium for engineering scenarios, so as to adapt to the localization needs and stringent rendering requirements of engineering documents.

[0012] To achieve the above objectives, the present invention employs the following technical means:

[0013] The first aspect of this invention proposes a formula recognition optimization method in engineering scenarios, comprising the following steps:

[0014] Construct a Chinese formula dataset based on the LaTeX seed formula dataset;

[0015] A formula recognition base model is constructed, and the base model is trained and optimized using the Chinese formula dataset. A rendering feedback mechanism is introduced during the training and optimization process to obtain an optimized formula recognition model. The rendering feedback mechanism renders the formula sequence predicted by the model as an image and compares it with the corresponding target image to obtain the rendering evaluation result, thereby guiding the training and optimization process of the model.

[0016] The formula image to be identified is input into the optimized formula recognition model for inference recognition. During the inference recognition process, the rendering feedback mechanism is applied to optimize the recognition result, and the final formula recognition result is output.

[0017] Preferably, the construction of the Chinese formula dataset based on the LaTeX seed formula dataset includes:

[0018] A Chinese element embedding corpus is constructed, which contains Chinese corpora at the word granularity and short sentence granularity, including basic general and rare character corpora and engineering field-specific corpora;

[0019] Obtain the LaTeX seed formula dataset, wherein the source code of the LaTeX seed formulas in the dataset is a token sequence that has undergone word segmentation.

[0020] LaTeX commands are pre-classified to guide subsequent strategies for embedding Chinese elements;

[0021] Based on the preset classification, the token sequence of the LaTeX seed formula in the dataset is traversed, and the corresponding Chinese element embedding strategy is adopted according to the instruction category to which the current token belongs, to obtain the first synthetic formula dataset.

[0022] Random text insertion is performed on the formulas in the first synthetic formula dataset with a preset probability to obtain the second synthetic formula dataset;

[0023] The LaTeX source code of the formulas in the second synthetic formula dataset is rendered as an image, and an image-source code pair is constructed to form the Chinese formula dataset.

[0024] Preferably, the preset classification includes independent instructions and non-independent instructions; the independent instructions include replaceable instructions and reserved instructions; and the non-independent instructions include reserved instructions, unary nested instructions, and binary nested instructions.

[0025] The concept of instruction blocks is introduced when processing the aforementioned non-independent instructions, wherein...

[0026] For reserved instructions and unary nested instructions, if they are immediately followed by curly braces, the instruction block includes the instruction and the first group of logical blocks enclosed by curly braces; otherwise, the instruction block includes the instruction and the single token that follows it.

[0027] For a nested binary instruction, if it is immediately followed by curly braces, the instruction block includes the instruction and the two subsequent logical blocks delimited by curly braces of the same level; otherwise, the instruction block includes the instruction and the two subsequent tokens.

[0028] Preferably, the Chinese element embedding strategy includes:

[0029] Iterate through the token sequence of the LaTeX seed formula sequentially and perform the following operations based on the instruction category to which the current token belongs:

[0030] If the current token belongs to a replaceable instruction in an independent instruction, it will be replaced with a Chinese element extracted from the corpus with a preset replacement probability.

[0031] If the current token belongs to a reserved instruction within a separate instruction, then the token is retained unchanged;

[0032] If the current token belongs to a reserved instruction in a non-independent instruction, then its corresponding complete instruction block is retained unchanged;

[0033] If the current token belongs to a unary nested instruction or a binary nested instruction in a non-independent instruction, then the token is retained, and the Chinese element embedding strategy is recursively executed on the corresponding one or two parameter parts within its instruction block.

[0034] Preferably, the step of performing random text insertion operation on the formulas in the first synthetic formula dataset with a preset probability includes:

[0035] When the formula is a single-line formula sequence, Chinese elements extracted from the corpus are inserted before the beginning or after the end of the formula sequence with a first preset probability.

[0036] When the formula is a multi-line formula sequence, Chinese elements extracted from the corpus are inserted before the beginning or after the end of each line of the formula sequence with a second preset probability.

[0037] Preferably, the construction of the formula recognition base model involves training and optimizing the base model using the Chinese formula dataset, and introducing a rendering feedback mechanism during the training and optimization process to obtain the optimized formula recognition model, including:

[0038] A formula recognition base model based on a visual encoder-decoder architecture is constructed, and the Chinese formula dataset is divided into a training set, a validation set, and a test set.

[0039] The base model is trained using the training set to obtain a baseline model with basic formula recognition capabilities.

[0040] Execute at least one round of training optimization loop based on rendering feedback mechanism on the baseline model using the training set until the model's performance evaluation result on the validation set no longer improves or the training optimization loop reaches a preset number of rounds. Each round of the loop includes:

[0041] The current model is used to perform sequence prediction on the training samples in the training set, and the prediction results are rendered as prediction images. A character detection matching algorithm is used to calculate the rendering quality score between the prediction image and the real image of the corresponding training sample.

[0042] The sampling weight of the corresponding training sample in the next training cycle is adjusted based on the rendering quality score, wherein the lower the rendering quality score, the higher the sampling weight of the corresponding training sample in the next training cycle.

[0043] The model is retrained by performing weighted random sampling from the training set based on the adjusted sampling weights, and its performance is evaluated on the validation set.

[0044] Preferably, the step of adjusting the sampling weights of the corresponding training samples in the next training cycle based on the rendering quality score is wherein the sampling weights are calculated according to the following formula:

[0045] ,

[0046] in, It is a sample Updated sampling weights It is a sample The rendering quality score, It is a focusing factor greater than 1, used to amplify the weight of low-scoring samples.

[0047] Preferably, the performance evaluation on the validation set includes evaluating the model's performance using a comprehensive evaluation metric, the formula for which the comprehensive evaluation metric is calculated is:

[0048] ,

[0049] in, For comprehensive evaluation indicators, For the original indicators, The average rendering quality score of the samples on the validation set is represented by α, which is a weighting parameter between 0 and 1.

[0050] Preferably, the step of inputting the formula image to be recognized into the optimized formula recognition model for inference recognition, and applying the rendering feedback mechanism to optimize the recognition result during the inference recognition process, and outputting the final formula recognition result includes:

[0051] Obtain the image of the formula to be recognized;

[0052] The image of the formula to be identified is preprocessed to obtain a standardized image;

[0053] The standardized image is input into the optimized formula recognition model, and a bundle search algorithm is used to generate at least two candidate formula sequences and their corresponding model log probabilities.

[0054] Each candidate formula sequence is rendered as a candidate image, and a character detection matching algorithm is used to calculate the rendering quality score between each candidate image and the standardized image.

[0055] The model log probability and rendering quality score of each candidate formula sequence are combined to calculate its fusion score, and all candidate formula sequences are reordered according to the fusion score.

[0056] The highest-ranking candidate formula sequence after reordering is output as the final formula recognition result.

[0057] Preferably, the fusion score is calculated by combining the model log probability and rendering quality score of each candidate formula sequence, and the formula for calculating the fusion score is:

[0058] ,

[0059] in, Candidate formula sequence The fusion score, Candidate formula sequence The model log probability, Candidate formula sequence Length, Candidate formula sequence The rendering quality score, It is a balanced hyperparameter between 0 and 1.

[0060] A second aspect of the present invention provides a formula recognition optimization device for engineering scenarios, comprising:

[0061] The dataset building module is used to build a Chinese formula dataset based on the LaTeX seed formula dataset;

[0062] The model training and optimization module is used to construct a formula recognition base model, train and optimize the base model using the Chinese formula dataset, and introduce a rendering feedback mechanism during the training and optimization process to obtain an optimized formula recognition model. The rendering feedback mechanism renders the formula sequence predicted by the model as an image and compares it with the corresponding target image to obtain a rendering evaluation result, thereby guiding the training and optimization process of the model.

[0063] The formula reasoning and recognition module is used to input the formula image to be recognized into the optimized formula recognition model for reasoning and recognition, and to apply the rendering feedback mechanism to optimize the recognition result during the reasoning and recognition process, and output the final formula recognition result.

[0064] A third aspect of the present invention provides an electronic device comprising a memory and a processor, wherein,

[0065] The memory is used to store programs;

[0066] The processor is used to execute the program to implement the above-described formula recognition optimization method in engineering scenarios.

[0067] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described formula recognition optimization method in an engineering scenario.

[0068] The beneficial effects of this invention are as follows:

[0069] (1) This invention solves the problems of data scarcity and domain adaptation. Through a data synthesis method based on LaTeX syntax features, it is possible to generate Chinese formula data with correct syntax, rich style and engineering terminology on a large scale. This lays the foundation for improving the ability of formula recognition models to recognize Chinese formulas, especially the coverage of rare Chinese characters and the accuracy of engineering terminology recognition.

[0070] (2) This invention improves training efficiency and model performance. By introducing an adaptive focusing training mechanism based on rendering feedback during training, the model can dynamically and purposefully learn difficult samples with low rendering scores, which accelerates the convergence of the model to complex and error-prone formula structures.

[0071] (3) This invention improves the availability and reliability of the recognition results. By introducing a cluster search mechanism based on rendering feedback in the reasoning, it combines the model’s internal probability prediction with objective, image-level rendering effectiveness, reduces the risk of outputting invalid or grammatically incorrect LaTeX sequences, significantly improves the rendering success rate of the formula, provides a high-quality and highly reliable data foundation for downstream applications, and has strong engineering practical value.

[0072] (4) The formula recognition optimization method provided by this invention has good universality, and its model training optimization and inference recognition optimization do not depend on the specific internal architecture of the formula recognition model. This method introduces an external feedback mechanism with rendering score as the core, which is applied to the training data input end and the model inference output end. In the training stage, the data sampling weight is dynamically adjusted through adaptive focusing training; in the inference stage, the candidate results are reordered through rendering-oriented beam search, thereby avoiding direct modification of the internal structure of the model. Therefore, the optimization method of this invention can be used as a general optimization method, applicable to the training and inference of most formula recognition models, thus broadening the application scope of this invention. Attached Figure Description

[0073] Figure 1 This is a schematic diagram of the overall process of the formula recognition optimization method in engineering scenarios provided in Embodiment 1 of the present invention;

[0074] Figure 2 This is a flowchart illustrating the process of constructing a Chinese formula dataset as provided in Embodiment 1 of the present invention;

[0075] Figure 3 This is a flowchart illustrating the training and optimization process of the formula recognition model provided in Embodiment 1 of the present invention;

[0076] Figure 4 This is a flowchart illustrating the reasoning and optimization process of the formula recognition model provided in Embodiment 1 of the present invention;

[0077] Figure 5 This is a LaTeX instruction classification diagram provided in Embodiment 1 of the present invention. Detailed Implementation

[0078] To make the objectives, techniques, and advantages of this invention clearer, the invention will be described clearly and completely below with reference to specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0079] Terminology Explanation:

[0080] CAD (Computer-Aided Design): refers to the technology of using computer software to assist in engineering design, drawing, analysis and optimization.

[0081] LaTeX (Lamport TeX): A document typesetting system and markup language based on TeX, particularly adept at handling complex mathematical formula typesetting. It is the standard text representation of the formulas in this patent; the seed dataset, model prediction targets, and final output all use LaTeX format.

[0082] Token: refers to the smallest indivisible unit with independent syntactic or semantic meaning extracted from the source text through tokenization.

[0083] CDM (Character Detection Matching): An image-level formula similarity measurement method that calculates similarity scores by detecting and matching characters and their spatial positions in two images. In this patent, it is used to evaluate the visual consistency between the rendered image of the predicted formula and the real image, serving as the basis for rendering feedback.

[0084] OCR (Optical Character Recognition): refers to the technology of converting text in an image into machine-encodeable text.

[0085] Transformer: A neural network architecture based on self-attention mechanism that can effectively process sequential data and model long-distance dependencies.

[0086] ViT (Vision Transformer): A neural network model that applies Transformer to image tasks. In this patent, it can be used as a visual encoder in a formula recognition model to encode the input formula image into a feature sequence.

[0087] Beam Search: A decoding algorithm for sequence generation. In the inference phase of this patent, it is used to generate multiple (k-beam widths) high-probability candidate LaTeX sequences.

[0088] BLEU (Bilingual Evaluation Understudy): An automatic evaluation metric based on n-gram matching: a metric used to evaluate sequence similarity, which calculates a score by comparing the n-gram overlap between the generated text and the reference text.

[0089] Edit Distance: A text similarity metric that measures the difference between two strings, expressed as the minimum number of single-character edits (insertions, deletions, replacements) required to transform one string into another.

[0090] Epoch (training round): In machine learning, an epoch refers to the process of a model completely traversing the entire training dataset for training. It is a basic unit of time in the training process.

[0091] Rendering: In this patent, it refers to the process of converting LaTeX sequence source code into a visual image.

[0092] Example 1

[0093] See Figure 1 This embodiment provides a formula recognition optimization method in engineering scenarios, which specifically includes the following steps:

[0094] S1, Construct a Chinese formula dataset based on the LaTeX seed formula dataset.

[0095] In this embodiment, step S1 provides a method for synthesizing Chinese formula data based on LaTeX seed formulas. Based on the LaTeX syntax characteristics, by performing syntax parsing and Chinese element embedding on the seed formulas without Chinese characters, a large-scale and diverse synthetic dataset is generated, and the synthetic dataset has a lexical preference for engineering scenarios through the Chinese word library in the engineering field, resulting in improved accuracy in the recognition of engineering terms and rare Chinese characters in the field. As Figure 2 shown, the construction of this Chinese formula dataset specifically includes the following steps:

[0096] S101. Construct a Chinese element embedding corpus, which contains Chinese corpora at the word granularity and short sentence granularity. The Chinese corpora include basic general and rare Chinese character corpora and engineering field-specific corpora.

[0097] Step S101 provides rich, accurate, and engineering-context-compliant Chinese embedding materials for subsequent formula synthesis by constructing a high-quality, multi-granularity, and engineering-field-expertise-rich Chinese corpus. This corpus contains Chinese corpora at the word granularity and short sentence granularity. Among them, the word granularity corpus is mainly used to embed independent semantic units such as variables, subscripts, and superscripts in the formula; the short sentence granularity corpus is mainly used to embed Chinese annotations, condition descriptions, or conjunctions beside the formula. To ensure the comprehensiveness and professionalism of the corpus, this embodiment mainly collects Chinese corpora through the following two ways:

[0098] One is the collection of basic general and rare Chinese character corpora. Specifically, high-frequency general Chinese vocabulary, common phrases, and related short sentences (such as "wherein", "let", "in summary") are extracted from various open-source text datasets such as encyclopedic corpora and news corpora, and at the same time, corpora containing rare Chinese characters and variant Chinese characters are collected to expand the coverage of characters and basic expression patterns.

[0099] The other is the collection of engineering field-specific corpora. Specifically, text content is extracted from open-source datasets and engineering field technical documents such as internal enterprise design specifications, standard manuals, technical reports, and drawing descriptions. First, engineering professional terms (such as "shearing force", "bending moment", "seepage coefficient") and typical expression short sentences (such as "according to the specification requirements", "see Appendix A for details") are preliminarily screened through methods such as word frequency statistics. Subsequently, professional personnel with engineering backgrounds conduct further manual review and screening of the candidate terms. Finally, a large number of engineering field professional terms and common engineering short sentences are collected.

[0100] By combining the above two ways, the finally constructed corpus combines generality and professionalism, laying a foundation for synthesizing a Chinese formula dataset that conforms to the engineering context, uses accurate words, and has natural sentence patterns.

[0101] S102, Obtain the LaTeX seed formula dataset. The source code of the LaTeX seed formula in the dataset is a token sequence that has been processed by word segmentation.

[0102] Specifically, this embodiment preferentially selects publicly available datasets such as IM2LATEX-100K and UniMER-1M as the seed formula dataset source. Each data point in these datasets is in the form of a formula image-LaTeX source code pair, and the LaTeX source code has already undergone pre-tokenization according to LaTeX syntax. Specifically, tokenization separates each LaTeX command, bracket, and text unit in the source code with spaces, thereby converting the continuous source code string into a structured token sequence. For example, after tokenization, the formula \frac{a}{b} has the token sequence \frac {a}{b}. This structured token sequence after tokenization allows for precise location of grammatical positions in the formula where Chinese characters can be embedded. Based on the corpus constructed in S101, Chinese elements that conform to LaTeX syntax rules can be replaced or inserted at these positions, avoiding syntax corruption or rendering failure caused by improper string processing.

[0103] S103 provides a pre-defined classification of LaTeX commands to guide subsequent strategies for embedding Chinese elements.

[0104] In step S103, the LaTeX instructions are processed according to their syntax functions as follows: Figure 5 The preset classification shown transforms the linear LaTeX token sequence into a structured object with clear semantics and operational rules, ensuring that subsequent Chinese elements can be accurately and compliantly embedded into the corresponding grammatical positions of the formula, thus avoiding damage to the LaTeX grammatical structure and rendering failure.

[0105] Specifically, based on whether instructions must take parameters, they are first divided into two main categories: independent instructions and non-independent instructions. Independent instructions are those that can exist independently syntactically and do not require a brace parameter, such as most mathematical operators, relational operators, and single letters. Non-independent instructions are those that syntactically require one or more parameters to form a complete semantic meaning, such as format control, functions, and structure commands.

[0106] Furthermore, based on their different roles in formula composition, the above two categories of instructions can be further subdivided. Independent instructions can be further divided into replaceable instructions and reserved instructions. Replaceable instructions are the primary target for Chinese embedding, including Greek letters (such as \alpha, \beta), English letters (such as A, x, y), and standard mathematical function names (such as \sin, \log). These instructions can be replaced with semantically compatible Chinese elements during embedding. The remaining operators, relational symbols, punctuation marks, etc., that constitute the mathematical skeleton of the formula (such as +, =) are classified as reserved instructions, which must remain unchanged during the Chinese formula composition process and are not replaced.

[0107] Furthermore, non-independent instructions can be further subdivided into reserved instructions, unary nested instructions, and binary nested instructions based on the number and function of their parameters. Reserved instructions (such as \mathbf{}, \mathrm{}) are instructions where both the command and parameters must be retained; unary nested instructions (such as \hat{}, \vec{}) are instructions that require only one parameter; and binary nested instructions (such as \frac{}{}, \binom{}{}) are instructions that require two parameters.

[0108] It should be noted that, to accurately handle non-independent instructions, this embodiment introduces the concept of an instruction block to define the scope of an instruction and its directly associated parameters. Specifically, for reserved instructions and unary nested instructions, if they are immediately followed by curly braces {}, their instruction block includes the instruction and the first set of logical blocks subsequently defined by curly braces; if they are not followed by curly braces, the instruction block includes the instruction and the single token that follows it. For binary nested instructions, if they are immediately followed by curly braces, their instruction block includes the instruction and the two sets of logical blocks subsequently defined by curly braces of the same level; if they are not followed by curly braces, the instruction block includes the instruction and the two tokens that follow it.

[0109] By establishing the above classification system and instruction block rules, we can accurately identify the role and scope of each syntactic unit in the formula, laying the foundation for the next step of implementing a differentiated, syntactically compliant Chinese element embedding strategy.

[0110] S104: Based on the preset classification, traverse the token sequence of LaTeX seed formulas in the dataset, and adopt the corresponding Chinese element embedding strategy according to the instruction category to which the current token belongs to obtain the first synthetic formula dataset.

[0111] This step, based on the LaTeX instruction classification system established by S103, performs reasonable and compliant Chinese element embedding operations on the token sequence of the seed formula while strictly adhering to the LaTeX syntax structure. Specifically, it sequentially traverses the token sequence of each LaTeX seed formula and adopts a differentiated embedding strategy based on the instruction category to which the current token belongs:

[0112] If the current token is a replaceable instruction in an independent instruction, it is decided whether to perform replacement with a preset replacement probability. Preferably, the replacement probability can be set to 0.15, and it can be specifically adjusted according to the Chinese density requirement of the synthesis formula. If replacement is triggered, a semantically adapted Chinese element is randomly selected from the corpus constructed in step S101 for replacement. For example, \sigma may be replaced with "stress", and \log may be replaced with "logarithm". Through probability constraints, while ensuring data diversity, the degree of Chineseization of the synthesis formula can be controlled to avoid the formula being difficult to understand due to excessive replacement.

[0113] If the current token is a reserved instruction in an independent instruction, the token remains unchanged. Such instructions (such as +, =, (, etc.) form the mathematical skeleton of the formula and need to be kept as they are to maintain the structural integrity of the formula.

[0114] If the current token is a reserved instruction in a non - independent instruction, the entire instruction block consisting of the instruction and all its associated parameters is retained without any internal replacement. For example, \mathbf{A} is retained as a whole.

[0115] If the current token is a unary nested instruction or a binary nested instruction in a non - independent instruction, first the instruction itself is retained, and then for one (unary) or two (binary) parameter parts within its instruction block, the embedding strategy of this step S104 is recursively executed, so that Chinese elements can be nested into complex formula structures.

[0116] The following uses the common bending normal stress formula "\sigma_{\text{max}}=\frac{M}{W}" in engineering as an example for illustration. Its corresponding LaTeX token sequence is "\sigma _ { \text { max}} = \frac { M}{ W}". When traversing this sequence, \sigma is recognized as a replaceable instruction and may be replaced with "stress" with the replacement probability. _ is recognized as a unary nested instruction, the instruction is retained, and its parameter {\text{max}} is recursively processed. During recursive processing, \text{max} is recognized as a reserved instruction in a non - independent instruction, so the entire instruction block is retained intact. = is recognized as a reserved instruction in an independent instruction and is retained. \frac is recognized as a binary nested instruction, the instruction is retained, and its two parameter blocks {M} and {W} are recursively processed separately. During recursive processing, M and W are recognized as replaceable instructions and may be replaced with "bending moment" and "flexural section modulus" respectively. Finally, the possible synthesized formula sequence is: stress _ { \text { max}} =\frac { bending moment} { flexural section modulus}.

[0117] S105, perform random text insertion operation on the formulas in the first synthetic formula dataset with a preset probability to obtain the second synthetic formula dataset.

[0118] It should be noted that in real engineering documents, formulas do not exist in isolation; they are often accompanied by explanatory, conditional, or related Chinese phrases or words before, after, or between lines, such as "where," "substituting into the above formula, we get:," "when x>0," etc. This mixed layout of text and formulas is a typical feature of engineering documents, especially Chinese engineering documents. However, existing recognition models based on public datasets are usually trained only on pure formula images and lack accurate understanding and recognition of this mixed layout pattern.

[0119] Based on this, this step designs a probability-controlled random text insertion strategy. On the basis of the synthesized formula sequence with embedded Chinese elements, additional Chinese elements randomly selected from the corpus constructed in step S101 are inserted at specific structural positions (such as the beginning and end of the formula, or line breaks in multi-line formulas). This allows for the synthesis of a large number of training samples containing contextual Chinese, enabling the model to recognize not only the mathematical structure and Chinese variables within the formula but also the annotation text outside the formula body during the learning process. Specifically, for any formula processed in step S104, this random text insertion strategy mainly includes two cases:

[0120] First, when the formula is a single-line formula sequence, a random number is generated. If the random number is less than a first preset probability (e.g., 0.1), an insertion operation is triggered. A short Chinese phrase or short sentence extracted from the corpus constructed in step S101 that conforms to the engineering context is randomly inserted before the start position (head) or after the end position (tail) of the formula sequence. Such phrases include "its calculation formula is:" or "in the formula". The insertion position (before the start position or after the end position) can be randomly selected with equal probability (e.g., 0.5 each). A colon, comma, or space can be automatically added as a separator between the inserted text and the formula according to the context. For example, for the synthetic formula "stress_{\text{max}} = \frac{bending moment}{bending section modulus}" generated in example S104, it is possible to insert "Bending normal stress calculation formula:" at the beginning to generate the final sequence "\text{bending normal stress calculation formula:} stress_{\text{max}} = \frac{bending moment}{bending section modulus}".

[0121] Secondly, for the insertion of multi-line formulas, for formulas containing multi-line environments such as array, cases, and aligned, a random number is first generated for each line of the formula sequence. If the random number is less than the second preset probability (e.g., 0.05), Chinese elements extracted from the above corpus are randomly inserted with equal probability (e.g., 0.5 each) before the beginning or after the end of the line sequence, such as "the first term represents:" or "solved:". This effectively simulates the common situation of annotating each line separately in formulas involving simultaneous equations, matrix descriptions, or case-by-case discussions.

[0122] Through the above-mentioned random text insertion strategy, the final dataset used to train the model not only contains isolated Chinese synthetic formulas, but also a large number of "formula-context text" hybrids that are close to engineering practice and have rich contextual information.

[0123] S106: Render the LaTeX source code of the formulas in the second synthetic formula dataset into images, construct image-source code pairs, and form a Chinese formula dataset.

[0124] In step S106, this embodiment preferably uses a rendering engine such as XeLaTeX that supports Chinese to render all the LaTeX source code sequences in the second synthetic formula dataset processed in step S105 into images, and constructs one-to-one image-source code data pairs, thereby finally forming a Chinese formula dataset with image and text pairs that can be used for model training and optimization.

[0125] S2. Construct a formula recognition base model, train and optimize the base model using a Chinese formula dataset, and introduce a rendering feedback mechanism during the training and optimization process to obtain an optimized formula recognition model. The rendering feedback mechanism renders the formula sequence predicted by the model as an image and compares it with the corresponding target image to obtain the rendering evaluation result, thereby guiding the training and optimization process of the model.

[0126] In this embodiment, step S2 provides a formula recognition training optimization method based on rendering feedback. By introducing a rendering feedback mechanism during the model training phase, the method increases attention to difficult samples during training, thereby improving the model's learning and fitting degree to difficult samples, and ultimately improving the grammatical correctness and rendering success rate of the recognition results. Specifically, this method uses the CDM (Character Detection Matching) algorithm as an evaluation metric for rendering quality. Unlike traditional text evaluation metrics such as edit distance, CDM is an image-level evaluation metric. CDM compares the input image with the image converted from the LaTeX formula predicted by the model, and uses visual feature extraction and localization techniques to perform precise character-level matching containing spatial location information, thus providing a more accurate and fairer evaluation result than traditional text-level metrics. Figure 3 As shown, the training and optimization method for the formula recognition model specifically includes the following steps:

[0127] S201, construct a formula recognition base model based on a visual encoder-decoder architecture, and divide the Chinese formula dataset into training set, validation set and test set.

[0128] Specifically, step S201 first selects and constructs a base model. In this embodiment, a formula recognition model based on a vision encoder-decoder architecture, such as the ViT + Transformer architecture, is preferred as the base model. The vision encoder is responsible for converting the input formula image into a series of semantically rich visual feature vectors. This embodiment preferably uses the Visual Transformer (ViT) or its variants as the encoder, whose self-attention mechanism can effectively model long-distance dependencies in the image, and is particularly suitable for capturing complex structural relationships between symbols in the formula, such as the correspondence between fractions, integrals, and parentheses. The decoder is responsible for autoregressively generating the target LaTeX token sequence based on the visual features output by the encoder. This embodiment preferably uses a standard Transformer decoder or a decoder optimized for the characteristics of mathematical formulas, utilizing its attention mechanism to dynamically focus on different regions of the image, thereby generating grammatically correct and symbolically accurate sequences.

[0129] Meanwhile, the Chinese formula dataset constructed in step S1 is randomly divided into a training set, a validation set, and a test set for subsequent training, optimization, and evaluation of the above formula recognition base model.

[0130] S202 uses the training set in the Chinese formula dataset to perform regular training on the base model, obtaining a baseline model with basic formula recognition capabilities.

[0131] This step uses standard supervised learning methods to initially train the constructed base model, obtaining a baseline model with basic recognition capabilities, laying the foundation for subsequent fine-grained optimization based on rendering feedback. Specifically, using the training set in the image-text pair Chinese formula dataset constructed in step S1, a large-scale data traversal is performed to learn the basic mapping relationship from formula images containing Chinese characters and context to the corresponding LaTeX sequences, thereby establishing a primary association between visual features and grammatical symbols. This embodiment uses a conventional training configuration, such as a standard, unweighted random sampler and cross-entropy loss function to train the model. This conventional training process is prior art, and its specific details are well known to those skilled in the art, and will not be elaborated here.

[0132] Furthermore, after the conventional training reaches stability, at least one round of training optimization based on the rendering feedback mechanism is performed on the baseline model using the training set. Each round of the loop includes:

[0133] S203: Use the current model to predict the sequence of training samples in the training set, render the prediction result as a prediction image, and use a character detection matching algorithm to calculate the rendering quality score between the prediction image and the real image of the corresponding training sample.

[0134] Specifically, after a training cycle, the current model is used to infer the samples in the training set to obtain the predicted LaTeX formula sequence. This predicted sequence is then re-rendered into an image using the same rendering environment as in step S106, resulting in a predicted rendered image. Next, the Character Detection Matching (CDM) algorithm is used to visually compare the predicted rendered image with the original real image (rendered from the tagged LaTeX source code) corresponding to the training samples. The CDM algorithm uses advanced visual feature extraction and character localization techniques to calculate the consistency of the two images in character content and spatial layout, ultimately providing a precise rendering quality score (CDM score), ranging from 0 to 1. A lower score indicates that although the predicted sequence may be close to the true value at the text level, its rendering result differs significantly from the target image; such samples are defined as "hard samples" under the current model. This step establishes an objective, rendering reliability-based measure of sample difficulty by calculating the CDM score for each training sample. It should be noted that a training cycle contains N epochs, where N can be chosen based on the degree of hard sample mining and the computational cost of CDM; for example, it can be set to 1 / 5 of the total number of training epochs.

[0135] S204, adjust the sampling weight of the corresponding training sample in the next training cycle based on the rendering quality score. The lower the rendering quality score, the higher the sampling weight of the corresponding training sample in the next training cycle.

[0136] Based on the CDM score calculated by S203, this step dynamically adjusts the sampling weight of each training sample in the next training cycle, thereby allowing the model to pay more attention to the "difficult samples" that are currently difficult to render correctly in subsequent training. Preferably, this embodiment uses a non-linear weight calculation formula:

[0137]

[0138] in, For the sample Updated sampling weights For the sample The rendering quality score, It is a focusing factor greater than 1 (e.g., it can be set to 2) used to amplify the weight of low-scoring samples.

[0139] This formula ensures differentiated weight adjustments. For easy samples with high rendering quality scores (close to 1), the weight increase is negligible; while for difficult samples with low rendering quality scores (close to 0), the weight is significantly amplified. For example, when At this point, the weight of a hard sample with a score of 0.3 will increase to approximately 1.49, while the weight of an easy sample with a score of 0.8 will only be 1.04. In this way, the data sampling strategy for subsequent training will be tilted towards hard samples, driving the model to perform targeted optimization.

[0140] S205, based on the adjusted sampling weights, performs weighted random sampling from the training set, retrains the current model, and evaluates its performance on the validation set.

[0141] Specifically, in a new training cycle, the data sampler performs weighted random sampling of the training set based on the updated sample weights to form training batches. Since hard samples have higher weights, their probability of being included in the batches increases significantly, greatly increasing the frequency with which the model encounters hard samples during training. Preferably, the training uses the same loss function as S202, but due to the change in data distribution, model gradient updates will be more driven by hard samples, thereby specifically improving model performance in error-prone and rendering failure-prone scenarios.

[0142] Subsequently, steps S203 to S205 are executed iteratively to form a closed-loop optimization. After each round of closed-loop optimization, the performance of the current model is evaluated on the validation set. Preferably, this embodiment uses a weighted index combining the original index and the CDM score as the final comprehensive evaluation index:

[0143]

[0144] in, For comprehensive evaluation indicators, For the original indicators, The average of the sample rendering quality scores (CDM scores) on the validation set. This is a weighting parameter between 0 and 1, used to adjust the contribution of the original metric and the CDM score to the final composite metric. The larger the value, the more the final metric is biased towards the original metric (optional 0.5). It should be noted that the original metric here refers to the metrics used to evaluate model performance in traditional formula recognition tasks, such as accuracy, edit distance, and BLEU scores commonly used in the field of formula recognition.

[0145] When the comprehensive evaluation index no longer improves significantly in several consecutive optimization iterations, or reaches the preset number of optimization iterations, the optimization process terminates, and the model with the highest comprehensive evaluation index on the validation set is selected as the final training optimization result.

[0146] The following example illustrates the execution process of steps S203 to S205:

[0147] Suppose the LaTeX sequence of the label is "\sqrt{\alpha}+\beta", the model might predict it as "\sqrt(\alpha)+\beta", which is an incorrect prediction of the brackets. Although the text edit distance is small, the predicted sequence fails to render or displays abnormally due to the mismatched brackets. In S203, the CDM score for this prediction will be extremely low. In S204, the weight of this sample is significantly increased. In the next training cycle in S205, the model will learn this sample more frequently due to weighted random sampling, thereby specifically correcting such syntax error patterns that easily lead to rendering failures. Through multiple such optimization cycles, the model gradually corrects the corresponding errors, ultimately improving its comprehensive evaluation metric on the validation set, which reflects both text accuracy and rendering success rate.

[0148] Through step S2, the final trained and optimized model is not only more accurate in character recognition, but also generates LaTeX code with extremely high syntactic accuracy and rendering success rate.

[0149] S3 inputs the formula image to be recognized into the optimized formula recognition model for inference recognition, and applies a rendering feedback mechanism to optimize the recognition results during the inference recognition process, and outputs the final formula recognition result.

[0150] Step S3 utilizes the optimized model trained in step S2 to perform actual inference on the formula images in the engineering document to be recognized and outputs a highly reliable recognition result. In this embodiment, step S3 provides a formula recognition inference optimization method based on rendering feedback. This method not only relies on the probability confidence of the candidate sequences generated by the model itself, but also innovatively introduces a character detection matching (CDM) rendering evaluation mechanism, which is identical to that used in the training phase, into the inference stage. This mechanism performs secondary screening and reordering of multiple candidate sequences, thereby ensuring that the final output LaTeX sequence has both high recognition confidence and high renderability. Figure 4 As shown, the reasoning and recognition optimization method specifically includes the following steps:

[0151] S301, Obtain the image of the formula to be recognized.

[0152] It should be noted that the formula images to be identified in this step are mainly formula images in engineering scenarios, which can come from various sources, including but not limited to scanned technical specifications, partial screenshots of drawings, formula illustrations in research reports, or images directly extracted from electronic documents such as PDFs.

[0153] S302, preprocess the formula image to be recognized to obtain a standardized image.

[0154] Specifically, this step performs standardization preprocessing on the formula image to be recognized, ensuring that its size, format, pixel distribution, and other aspects are consistent with the data specifications used during model training, thus meeting the requirements of the model input. Preprocessing operations include, but are not limited to, converting the image to a uniform color mode (such as grayscale or RGB), adjusting the image resolution, normalizing the size, and normalizing the pixel values.

[0155] S303, input the standardized image into the optimized formula recognition model, and use the beam search algorithm to generate at least two candidate formula sequences and their corresponding model log probabilities.

[0156] Specifically, the model first extracts depth features from the image using a visual encoder, and then the decoder generates a LaTeX token sequence in an autoregressive manner based on these features. To capture multiple possible reasonable recognition results, rather than outputting only a single optimal sequence, this step preferably uses a beam search algorithm for decoding, setting the beam size to [value missing]. (For example The algorithm retains the k candidate sequences with the highest probabilities at each generation step, and continues to expand the next token based on these candidates, ultimately outputting k complete candidate LaTeX sequences and their corresponding model log probabilities. These candidate sequences all have high model confidence at the text level, but some of them may contain minor grammatical errors (such as mismatched brackets or misused commands) or semantic biases.

[0157] S304, render each candidate formula sequence as a candidate image, and use a character detection matching algorithm to calculate the rendering quality score between each candidate image and the normalized image.

[0158] Specifically, in this step, the k candidate LaTeX sequences generated in step S303 are first fed in parallel or serially into a LaTeX rendering environment consistent with steps S106 and S203, preferably XeLaTeX, to render and generate k corresponding candidate images. Then, the CDM algorithm is used to visually compare each candidate image with the standardized image preprocessed in step S302. By analyzing the consistency of character content, shape, and spatial layout, a quantified rendering quality score is calculated for each candidate sequence, with a value between 0 and 1. A higher score indicates better visual fidelity. Finally, a set of rendering scores for all candidate sequences is obtained. .

[0159] S305, calculate the fusion score of each candidate formula sequence by combining the model log probability and the rendering quality score, and reorder all candidate formula sequences according to the fusion score.

[0160] It should be understood that candidate sequences with high model log probabilities do not necessarily have good rendering effects, and vice versa. Therefore, this step fuses the model log probability and rendering quality score of each candidate sequence to obtain a fusion score that balances text and visual aspects, and makes the optimal decision based on this fusion score.

[0161] Specifically, for each candidate sequence Its fusion score The calculation formula is:

[0162]

[0163] in, Candidate sequences The model's logarithmic probability; Candidate sequences The length of is used for length normalization; It is a balancing hyperparameter between 0 and 1 (e.g.) ), used to adjust the weights of model confidence and rendering effectiveness; Candidate sequences The rendering quality score.

[0164] Calculate the fusion score of all candidate sequences. Then, based on the fusion score... The candidate sequences are sorted in descending order. The sequence ranked first is the one that achieves the best balance between model confidence and rendering reliability.

[0165] S306 outputs the candidate formula sequence with the highest ranking after reordering as the final formula recognition result.

[0166] In the output stage, the candidate LaTeX sequence that ranks first after reordering in step S305 is selected as the final recognition result with both high confidence and high renderability.

[0167] The following is an illustrative description of the execution process of step S3.

[0168] Suppose we input an image of a formula. The model generates 5 candidate sequences using a beam search algorithm, among which the two main candidate sequences are A: "\frac{a}{b+c}", and the model's log probability is... High; B: "\frac{a}{b}+c", its model log probability Slightly lower. Based solely on the model's logarithmic probability, it might output A. However, after CDM rendering scoring, it was found that the rendered image of B is almost identical to the input image (CDM score close to 1.0), while the rendered image of A has a fractional structure error due to the lack of necessary parentheses (lower CDM score). In step S305, B's fusion score may surpass A due to its extremely high CDM score, thus being correctly output as the final result.

[0169] Example 2

[0170] This embodiment provides a formula recognition optimization device for engineering scenarios, including:

[0171] The dataset building module is used to build a Chinese formula dataset based on the LaTeX seed formula dataset;

[0172] The model training and optimization module is used to construct a formula recognition base model, train and optimize the base model using the Chinese formula dataset, and introduce a rendering feedback mechanism during the training and optimization process to obtain an optimized formula recognition model. The rendering feedback mechanism renders the formula sequence predicted by the model as an image and compares it with the corresponding target image to obtain a rendering evaluation result, thereby guiding the training and optimization process of the model.

[0173] The formula reasoning and recognition module is used to input the formula image to be recognized into the optimized formula recognition model for reasoning and recognition, and to apply the rendering feedback mechanism to optimize the recognition result during the reasoning and recognition process, and output the final formula recognition result.

[0174] Furthermore, this embodiment also provides an electronic device, including a memory and a processor, wherein,

[0175] The memory is used to store programs;

[0176] The processor is used to execute the program to implement the formula recognition optimization method in the engineering scenario described in Embodiment 1.

[0177] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the formula recognition optimization method in the engineering scenario described in Embodiment 1.

Claims

1. A formula recognition optimization method in engineering scenarios, characterized in that, Includes the following steps: Construct a Chinese formula dataset based on the LaTeX seed formula dataset; A formula recognition base model is constructed, and the base model is trained and optimized using the Chinese formula dataset. A rendering feedback mechanism is introduced during the training and optimization process to obtain an optimized formula recognition model. The rendering feedback mechanism renders the formula sequence predicted by the model as an image and compares it with the corresponding target image to obtain a rendering quality score. The sampling weights of the training samples are dynamically adjusted based on the rendering quality score, and the training set is weighted and sampled according to the adjusted sampling weights for model retraining. The formula image to be identified is input into the optimized formula recognition model for inference recognition. During the inference recognition process, the rendering feedback mechanism is applied to optimize the recognition result and output the final formula recognition result. The rendering feedback mechanism is used to calculate the rendering quality score for each of the multiple candidate formula sequences generated by the model based on the beam search algorithm, and to perform a fusion score by combining the model log probability of each candidate sequence with the rendering quality score. The order of the candidate formula sequences is adjusted based on the fusion score.

2. The formula recognition optimization method in engineering scenarios according to claim 1, characterized in that, The construction of the Chinese formula dataset based on the LaTeX seed formula dataset includes: A Chinese element embedding corpus is constructed, which contains Chinese corpora at the word level and the sentence level, including basic general and rare character corpora and engineering-specific corpora; Obtain the LaTeX seed formula dataset, wherein the source code of the LaTeX seed formulas in the dataset is a token sequence that has undergone word segmentation. LaTeX commands are pre-classified to guide subsequent strategies for embedding Chinese elements; Based on the preset classification, the token sequence of the LaTeX seed formula in the dataset is traversed, and the corresponding Chinese element embedding strategy is adopted according to the instruction category to which the current token belongs, to obtain the first synthetic formula dataset. Random text insertion is performed on the formulas in the first synthetic formula dataset with a preset probability to obtain the second synthetic formula dataset; The LaTeX source code of the formulas in the second synthetic formula dataset is rendered as an image, and an image-source code pair is constructed to form the Chinese formula dataset.

3. The formula recognition optimization method in engineering scenarios according to claim 2, characterized in that, The preset classification includes independent instructions and non-independent instructions. The independent instructions include replaceable instructions and reserved instructions. The non-independent instructions include reserved instructions, unary nested instructions, and binary nested instructions. The concept of instruction blocks is introduced when processing the aforementioned non-independent instructions, wherein... For reserved instructions and unary nested instructions, if they are immediately followed by curly braces, the instruction block includes the instruction and the first group of logical blocks enclosed by curly braces; otherwise, the instruction block includes the instruction and the single token that follows it. For a nested binary instruction, if it is immediately followed by curly braces, the instruction block includes the instruction and the two subsequent logical blocks delimited by curly braces of the same level; otherwise, the instruction block includes the instruction and the two subsequent tokens.

4. The formula recognition optimization method in engineering scenarios according to claim 3, characterized in that, The Chinese element embedding strategy includes: Iterate through the token sequence of the LaTeX seed formula sequentially and perform the following operations based on the instruction category to which the current token belongs: If the current token belongs to a replaceable instruction in an independent instruction, it will be replaced with a Chinese element extracted from the corpus with a preset replacement probability. If the current token belongs to a reserved instruction within a separate instruction, then the token is retained unchanged; If the current token belongs to a reserved instruction in a non-independent instruction, then its corresponding complete instruction block is retained unchanged; If the current token belongs to a unary nested instruction or a binary nested instruction in a non-independent instruction, then the token is retained, and the Chinese element embedding strategy is recursively executed on the corresponding one or two parameter parts within its instruction block.

5. The formula recognition optimization method in engineering scenarios according to claim 2, characterized in that, The step of performing random text insertion operation on the formulas in the first synthetic formula dataset with a preset probability includes: When the formula is a single-line formula sequence, Chinese elements extracted from the corpus are inserted before the beginning or after the end of the formula sequence with a first preset probability. When the formula is a multi-line formula sequence, Chinese elements extracted from the corpus are inserted before the beginning or after the end of each line of the formula sequence with a second preset probability.

6. The formula recognition optimization method in engineering scenarios according to claim 1, characterized in that, The construction of the formula recognition base model involves training and optimizing the base model using the Chinese formula dataset, and introducing a rendering feedback mechanism during the training and optimization process to obtain the optimized formula recognition model, including: A formula recognition base model based on a visual encoder-decoder architecture is constructed, and the Chinese formula dataset is divided into a training set, a validation set, and a test set. The base model is trained using the training set to obtain a baseline model with basic formula recognition capabilities. Execute at least one round of training optimization loop based on rendering feedback mechanism on the baseline model using the training set until the model's performance evaluation result on the validation set no longer improves or the training optimization loop reaches a preset number of rounds. Each round of the loop includes: The current model is used to perform sequence prediction on the training samples in the training set, and the prediction results are rendered as prediction images. A character detection matching algorithm is used to calculate the rendering quality score between the prediction image and the real image of the corresponding training sample. The sampling weight of the corresponding training sample in the next training cycle is adjusted based on the rendering quality score, wherein the lower the rendering quality score, the higher the sampling weight of the corresponding training sample in the next training cycle. The model is retrained by performing weighted random sampling from the training set based on the adjusted sampling weights, and its performance is evaluated on the validation set.

7. The formula recognition optimization method in engineering scenarios according to claim 6, characterized in that, The sampling weights of the corresponding training samples in the next training cycle are adjusted based on the rendering quality score, wherein the sampling weights are calculated according to the following formula: ， in, It is a sample Updated sampling weights It is a sample The rendering quality score, It is a focusing factor greater than 1, used to amplify the weight of low-scoring samples.

8. The formula recognition optimization method in engineering scenarios according to claim 6 or 7, characterized in that, The performance evaluation on the validation set includes evaluating the model's performance using a comprehensive evaluation metric, the formula for which the comprehensive evaluation metric is calculated is: ， in, For comprehensive evaluation indicators, For the original indicators, The average rendering quality score of the samples on the validation set is represented by α, which is a weighting parameter between 0 and 1.

9. The formula recognition optimization method in engineering scenarios according to claim 1, characterized in that, The process of inputting the formula image to be recognized into the optimized formula recognition model for inference recognition, and applying the rendering feedback mechanism to optimize the recognition result during the inference recognition process, and outputting the final formula recognition result includes: Obtain the image of the formula to be recognized; The image of the formula to be identified is preprocessed to obtain a standardized image; The standardized image is input into the optimized formula recognition model, and a bundle search algorithm is used to generate at least two candidate formula sequences and their corresponding model log probabilities. Each candidate formula sequence is rendered as a candidate image, and a character detection matching algorithm is used to calculate the rendering quality score between each candidate image and the standardized image. The model log probability and rendering quality score of each candidate formula sequence are combined to calculate its fusion score, and all candidate formula sequences are reordered according to the fusion score. The highest-ranking candidate formula sequence after reordering is output as the final formula recognition result.

10. The formula recognition optimization method in engineering scenarios according to claim 9, characterized in that, The fusion score is calculated by combining the model log probability and rendering quality score of each candidate formula sequence. The formula for calculating the fusion score is as follows: ， in, Candidate formula sequence The fusion score, Candidate formula sequence The model log probability, Candidate formula sequence Length, Candidate formula sequence The rendering quality score, It is a balanced hyperparameter between 0 and 1.

11. A formula recognition and optimization device for engineering scenarios, characterized in that, include: The dataset building module is used to build a Chinese formula dataset based on the LaTeX seed formula dataset; The model training and optimization module is used to construct a formula recognition base model, train and optimize the base model using the Chinese formula dataset, and introduce a rendering feedback mechanism during the training and optimization process to obtain an optimized formula recognition model. The rendering feedback mechanism renders the formula sequence predicted by the model as an image and compares it with the corresponding target image to obtain a rendering quality score. Based on the rendering quality score, the sampling weights of the training samples are dynamically adjusted, and the training set is weighted and sampled according to the adjusted sampling weights for model retraining. The formula reasoning and recognition module is used to input the formula image to be recognized into the optimized formula recognition model for reasoning and recognition, and to apply the rendering feedback mechanism to optimize the recognition result during the reasoning and recognition process, and output the final formula recognition result; wherein, the rendering feedback mechanism is used to calculate the rendering quality score for each of the multiple candidate formula sequences generated by the model based on the beam search algorithm, and to combine the model log probability of each candidate sequence with the rendering quality score to perform a fusion score, and to adjust the order of the candidate formula sequences based on the fusion score.

12. An electronic device, characterized in that, Including memory and processor, among which, The memory is used to store programs; The processor is used to execute the program to implement the formula recognition optimization method in engineering scenarios as described in any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the formula recognition optimization method for engineering scenarios as described in any one of claims 1-10.

Images