High-precision handwritten mathematical formula recognition methods, systems, devices, and storage media

Abstract

This invention discloses a high-precision handwritten mathematical formula recognition method, system, device, and storage medium. These are corresponding solutions. Addressing the problem that existing recognition methods over-rely on text-level evaluation metrics, leading to low visual fidelity and an inability to effectively handle structured ambiguities and subtle structural errors, this invention establishes a mapping from the symbolic domain to the visual domain. It measures the image-level matching score by calculating the difference between the predicted formula image and the true formula image, and uses this score as a reward signal for network training, directly updating the parameters of the policy network. This effectively overcomes the problem of multiple codes for a single image, significantly improving the model's accuracy and robustness in recognizing the structure of complex mathematical expressions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and pattern recognition, and in particular to a high-precision handwritten mathematical formula recognition method, system, device and storage medium. BACKGROUND

[0002] Handwritten Mathematical Expression Recognition (HMER) is an important research direction in the field of computer vision and artificial intelligence, widely applied in document digitization, intelligent education, office automation and other fields. With the development of deep learning technology, the research of HMER has made significant progress, especially in the recognition of complex mathematical expressions, many deep learning models based on convolutional neural networks (CNN), recurrent neural networks (RNN) and Transformer have been proposed and achieved certain results. However, although the existing technology has improved the recognition accuracy, there are still deficiencies in the optimization of complex structure and visual consistency.

[0003] Currently, the handwritten mathematical formula recognition method can be mainly divided into tree structure-based method and sequence-based method. The tree structure-based method recognizes by analyzing the syntax structure of mathematical formulas, but it has poor flexibility in handling complex formulas and symbol layout, and is difficult to cope with diversified handwriting styles. The sequence method regards handwritten mathematical formulas as an image-to-text conversion problem, and uses an end-to-end neural network model to directly output LaTeX (a markup language) code, which has strong adaptability.

[0004] However, the existing text-level evaluation method has certain limitations. First, different LaTeX syntax can generate the same rendered image, resulting in incorrect judgment of text-level evaluation, which cannot fully capture the visual differences between rendered images. Second, small text errors (such as symbol position errors or symbol errors) can have a serious impact on the visual effect of the formula, but the text-level evaluation index punishes such errors lightly, failing to effectively guide model optimization. In addition, traditional text similarity-based training methods perform poorly when handling complex structure handwritten formulas, especially in the spatial arrangement and structural relationship between symbols. However, traditional methods often rely on text-level similarity evaluation standards. In the maximum likelihood estimation (MLE) framework, existing methods optimize by calculating the cross-entropy of text pairs; while in the reinforcement learning framework, they rely on text-level evaluation standards such as BLEU and ROUGE. These methods fail to effectively consider the text and visual consistency of formulas, making it difficult for the predicted LaTeX code to accurately restore the visual effect of handwritten formulas.

[0005] Therefore, existing technologies have significant shortcomings in handwritten mathematical formula recognition, especially in optimizing formula visual consistency and symbol accuracy, necessitating a new method to address these issues. Current technology lacks an optimization scheme that comprehensively considers formula structure and visual representation, leading to certain accuracy and stability problems in handwritten mathematical formula recognition.

[0006] In view of this, the present invention is hereby proposed. Summary of the Invention

[0007] The purpose of this invention is to provide a high-precision handwritten mathematical formula recognition method, system, device, and storage medium, which can significantly improve the visual fidelity and structural accuracy of handwritten mathematical formula recognition.

[0008] The objective of this invention is achieved through the following technical solution: A high-precision method for recognizing handwritten mathematical formulas includes: Construct a policy network; The policy network is trained using the training set, including: performing feature encoding and decoding sampling on handwritten mathematical formula images in the training set based on the policy network to generate several candidate LaTeX character sequences; transforming each candidate LaTeX character sequence and the corresponding ground truth LaTeX character sequence in the training set using a rendering engine to obtain several predicted formula images and one ground truth formula image; calculating the image-level matching degree between each predicted formula image and the ground truth formula image, using the image-level matching degree as the reward value for reinforcement learning, and updating the policy network using a group relative policy optimization method; After training, the handwritten mathematical formula image to be recognized is input into the trained policy network to obtain candidate LaTeX character sequences, which are then converted into predicted formula images by the rendering engine.

[0009] A high-precision handwritten mathematical formula recognition system, used to implement the aforementioned method, includes: Network building blocks are used to build policy networks; The network training unit is used to train the policy network using the training set. This includes: performing feature encoding and decoding sampling on handwritten mathematical formula images in the training set based on the policy network to generate several candidate LaTeX character sequences; transforming each candidate LaTeX character sequence and the corresponding ground truth LaTeX character sequence in the training set using a rendering engine to obtain several predicted formula images and one ground truth formula image; calculating the image-level matching degree between each predicted formula image and the ground truth formula image, using the image-level matching degree as the reward value for reinforcement learning, and updating the policy network using a group relative policy optimization method. The recognition unit is used to input the handwritten mathematical formula image to be recognized into the trained policy network after training, obtain candidate LaTeX character sequences, and convert them into predicted formula images through the rendering engine.

[0010] A processing device includes: one or more processors; and a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0011] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0012] As can be seen from the technical solution provided by the present invention, existing recognition methods rely too heavily on text-level evaluation metrics, resulting in low visual fidelity of recognition results and an inability to effectively handle structured ambiguity and subtle structural errors. The solution provided by the present invention establishes a mapping from the symbolic domain to the visual domain, measures the image-level matching score by calculating the degree of difference between the predicted formula image and the true formula image, and uses it as a reward signal for network training to directly update the parameters of the policy network. This effectively overcomes the problem of multiple codes for a single image and significantly improves the model's accuracy and robustness in structural recognition of complex mathematical expressions. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart illustrating a high-precision handwritten mathematical formula recognition method provided in an embodiment of the present invention;

[0015] Figure 2 This is a schematic diagram of the overall framework of the policy network training process provided in an embodiment of the present invention.

[0016] Figure 3 This is a schematic diagram of a high-precision handwritten mathematical formula recognition system provided in an embodiment of the present invention.

[0017] Figure 4 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0019] First, the following explanations are provided for the terms that may be used in this article: The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0020] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0021] The high-precision handwritten mathematical formula recognition method, system, device, and storage medium provided by this invention are described in detail below. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they are performed according to conventional conditions in the art or conditions recommended by the manufacturer. Instruments used in the embodiments of this invention whose manufacturers are not specified are all conventional products that can be purchased commercially.

[0022] Example 1 This invention provides a high-precision handwritten mathematical formula recognition method, such as... Figure 1 As shown, the main steps include the following: Step 1: Construct the policy network.

[0023] In this embodiment of the invention, a policy network based on an encoder-decoder structure is constructed.

[0024] Step 2: Train the policy network using the training set.

[0025] In this embodiment of the invention, a policy network is used to perform feature encoding and decoding sampling on handwritten mathematical formula images in the training set to generate several candidate LaTeX character sequences. Each candidate LaTeX character sequence and the corresponding ground truth LaTeX character sequence in the training set are then transformed by a rendering engine to obtain several predicted formula images and one ground truth formula image. The image-level matching degree between each predicted formula image and the ground truth formula image is calculated, and the image-level matching degree is used as the reward value for reinforcement learning. A group relative policy optimization method is then employed to update the policy network.

[0026] In this embodiment of the invention, calculating the image-level matching degree between each predicted formula image and the ground truth formula image includes: preprocessing each predicted formula image and the ground truth formula image respectively; converting each preprocessed predicted formula image and the ground truth formula image into corresponding one-dimensional integer sequences respectively; calculating the Levenstein edit distance of the one-dimensional integer sequence corresponding to each predicted formula image and the one-dimensional integer sequence corresponding to the ground truth formula image, and normalizing it according to the maximum length of the one-dimensional integer sequence to obtain an image matching score, which is used as the image-level matching degree.

[0027] In this embodiment of the invention, the preprocessing of each predicted formula image and the ground truth formula image includes: filling the data matrices of each predicted formula image and the ground truth formula image to the same height dimension and keeping the content vertically centered; then, using a fixed threshold, performing binarization processing on the filled data matrices respectively, setting the pixel value to 1 if it is greater than the fixed threshold, and setting it to 0 otherwise; finally, scanning and removing all column vectors in each image data matrix where the pixel value is completely 0.

[0028] In this embodiment of the invention, the formula for calculating the image matching score is as follows: ; in, Let G be the one-dimensional integer sequence corresponding to the truth formula image G. For a single predicted formula image P, there is a one-dimensional integer sequence. A one-dimensional integer sequence and Lewinstein's edit distance; and Represent one-dimensional integer sequences respectively and Length; To predict the image matching score between the formula image P and the true formula image G.

[0029] In this embodiment of the invention, when training the policy network, the policy network is initialized. Simultaneously instantiate the old policy network. and reference policy network Furthermore, during the training of the policy network, the old policy network is periodically updated based on the parameters of the updated policy network. and reference policy network The parameters; the feature encoding and decoding sampling of handwritten mathematical formula images in the training set based on the policy network to generate several candidate LaTeX character sequences refers to using the old policy network that is periodically updated based on the policy network. Feature encoding and decoding sampling are performed on the handwritten mathematical formula images in the training set to generate several candidate LaTeX character sequences.

[0030] The policy network is updated as follows: based on image-level matching degree, a group relative policy optimization method is used to calculate the advantage value corresponding to each candidate LaTeX character sequence; and, based on the policy network... With the old policy network Calculate the importance sampling ratio; combine the importance sampling ratio, dominance value, and cutoff parameter to calculate the alternative target term; based on the policy network. With reference policy network Calculate the KL divergence regularization term; construct a loss function by combining the alternative objective term and the KL divergence regularization term, and update the policy network through the loss function.

[0031] In this embodiment of the invention, the calculation of the advantage value corresponding to each candidate LaTeX character sequence using the group relative strategy optimization method based on image-level matching degree includes: For a single handwritten mathematical formula image, N candidate LaTeX character sequences are generated; for the i-th candidate LaTeX character sequence, the image-level matching degree between its corresponding predicted formula image and the true formula image is denoted as . Advantage value Calculated using the following formula: ; in, Let be the statistical mean of the image-level matching degree (i.e., image matching score) of N candidate LaTeX character sequences. is the statistical standard deviation of the image-level matching degree (i.e., image matching score) of N candidate LaTeX character sequences.

[0032] In this embodiment of the invention, the loss function is expressed as: ; in, Let be the loss function, and be the mathematical expectation. To replace the target item, For KL divergence regularization, Indicates the importance sampling ratio, where Represents a sequence of historical LaTeX characters. Generate current tag The conditional state, and Representing policy networks With the old policy network Based on historical LaTeX character sequences Generate current tag The probability distribution of the two is used to measure and constrain the magnitude of policy updates; This represents the truncation function. To truncate parameters; Denotes KL divergence, For KL coefficients.

[0033] Step 3: Use the trained policy network to achieve high-precision recognition of handwritten mathematical formulas.

[0034] In this embodiment of the invention, after training is completed, the handwritten mathematical formula image to be recognized is input into the trained policy network to obtain candidate LaTeX character sequences, which are then converted into predicted formula images by the rendering engine.

[0035] The above-described solution provided by the embodiments of the present invention mainly achieves the following beneficial effects: (1) Direct optimization at the visual level is achieved, solving the problem of multiple codes for one image: This invention overcomes the shortcomings of traditional text-level metrics (such as BLEU and ROUGE) in accurately reflecting the visual structure of mathematical formulas by evaluating the visual level after rendering. This method can effectively identify multiple codes for one image with different text sequences but the same rendering effect, avoid unreasonable error penalties, and ensure that the model focuses on restoring the visual truth value of the formula.

[0036] (2) Significantly improves the recognition accuracy of complex two-dimensional structures: The image matching evaluation mechanism proposed in this invention, through image-level structural feature comparison, not only preserves the horizontal layout information of the formula, but also captures the fine structure in the vertical direction. This mechanism is highly sensitive to subtle structural errors such as misaligned superscripts and subscripts and confusion of symbols commonly found in mathematical formulas, thus exhibiting higher accuracy when processing mathematical expressions with complex nested structures.

[0037] (3) Enhanced robustness and generalization ability of evaluation indicators: Before evaluation, the present invention removes interference from irrelevant backgrounds through preprocessing, making the reward signal invariant to global horizontal displacement and focusing on the structural alignment of content. Experiments show that this visual matching-based optimization method can guide the recognition system to maintain excellent stability under different resolutions and noise interference.

[0038] (4) The training architecture is efficient and highly compatible: This invention utilizes the GRPO framework to eliminate the need for independent value networks in traditional reinforcement learning, significantly reducing memory usage and computational resource consumption, and simplifying the training process. At the same time, as a general optimization strategy, it does not depend on a specific backbone network structure and can be seamlessly integrated into various existing sequence generation recognition models, thus having wide applicability.

[0039] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0040] I. Overall Description of the Plan

[0041] This invention provides a high-precision method for recognizing handwritten mathematical formulas. The method first acquires images of handwritten mathematical formulas from a training set, and then uses a policy network-based recognition model to perform feature encoding and decoding sampling on these images, generating several candidate LaTeX character sequences. Next, a mapping from the symbolic domain to the visual domain is established, and a rendering engine converts the candidate LaTeX character sequences and their corresponding ground truth LaTeX character sequences into visualized predicted and ground truth formula images, respectively. Based on this, the invention constructs an Image Matching Score (IMS), which quantifies the fidelity of the predicted and ground truth formula images in a two-dimensional visual layout by calculating the image-level matching degree between them. Finally, the image matching score is directly used as a reward signal for reinforcement learning, and the Group Relative Policy Optimization (GRPO) method is employed to update the parameters of the policy network, guiding the model to optimize towards generating sequences with higher visual consistency.

[0042] II. Detailed introduction of the plan.

[0043] 1. Problem Definition and Formalization

[0044] To clarify the processing object and objective of this invention, the recognition task is first formally defined. Given an input image of a handwritten mathematical expression... ,in, Let H be the set of real numbers, and let W represent the height and width of the image, respectively. The goal of this invention is to generate a symbol sequence. Each of them It belongs to the predefined LaTeX vocabulary C (e.g., `\frac`, `\sin`, `a`, `b`, etc.), and T represents the length of the sequence.

[0045] The core of this invention lies in optimizing the policy network so that the visual image of the generated symbol sequence Y after rendering is highly consistent with the input image X (or its ground truth image) in terms of visual structure.

[0046] 2. Overall process introduction.

[0047] (1) Construct and initialize the policy network.

[0048] First, a policy network based on an encoder-decoder architecture is constructed and configured as the core recognition model. In this embodiment, the encoder (preferably using a DenseNet-121 backbone network) is configured to perform the feature extraction task, transforming the input handwritten mathematical formula image into a two-dimensional feature map; simultaneously, the decoder (preferably using a Transformer architecture with a multi-head self-attention mechanism) is configured to receive the feature map and autoregressively generate LaTeX character sequences.

[0049] During the training startup phase, the policy network is initialized. parameter Synchronously instantiate the old policy network and reference policy network Assign their initial parameters to values ​​that are the same as those of the previous ones. The same principle applies, providing a benchmark for subsequent strategy iterations.

[0050] In this embodiment of the invention, the entire scheme can be completed based on a computer. The policy network can be built in memory, and the corresponding old policy network and reference policy network are both configured in the computer's storage space.

[0051] (2) Sampling of candidate sequences based on the current strategy.

[0052] In each training iteration, the handwritten mathematical formula image data stream from the training set is input into the old policy network. In this embodiment of the invention, Monte Carlo sampling logic is used to control the old policy network to perform multiple forward inferences on the same input image signal, generating N candidate LaTeX character sequences in parallel.

[0053] For example, the quantity N is set to 16, and the sampling temperature parameter is configured. By increasing the randomness of the generated sequences, the system is able to explore a wider prediction space.

[0054] (3) Calculate the image matching score (IMS).

[0055] To feed back the visual fidelity to the optimization module, the rendering engine and visual processing module are invoked to calculate the Image Matching Score (IMS) for each candidate sequence as the image-level matching degree. The term "image matching score" will be used throughout the following description. The specific processing logic is as follows:

[0056] First, the pre-built LaTeX rendering engine is invoked to convert each candidate LaTeX character sequence into corresponding predicted formula image data, while the corresponding ground truth LaTeX sequence is converted into ground truth formula image data. To eliminate interference from differences in image format, the visual processing module performs unified preprocessing operations on the predicted and ground truth formula images: the data matrices of the two formula images are padded to the same height dimension while keeping the content vertically centered; the pixel data (i.e., the padded data matrix) is binarized using a fixed threshold (e.g., 128), setting a pixel value greater than the fixed threshold to 1 and otherwise to 0; finally, all column vectors with completely zero pixel values ​​in each image data matrix are scanned and removed to eliminate the impact of global horizontal displacement on matching accuracy.

[0057] Secondly, the preprocessed two-dimensional image data is transformed into a one-dimensional integer feature sequence. Specifically, for each column in the image data matrix, it is mapped to a binary vector and further converted into a corresponding decimal integer value. Through this step, the predicted formula image is encoded into an integer sequence. The truth formula image is encoded as a sequence of integers. .

[0058] Finally, calculate the sequence. with sequence The Levenshtein distance between them is denoted as... And based on the maximum length of the two sequences The distance is then normalized. Specifically, the formula for calculating the image matching score is as follows: ; Where F represents the prediction formula image and G represents the truth formula image. and These represent the lengths of the corresponding integer sequences. The score ranges from [0, 1] and serves as a quantitative indicator to measure the accuracy of the visual structure generated by the system.

[0059] (4) Estimation of the advantage function.

[0060] The Group Relative Policy Optimization (GRPO) logic is executed, which does not require maintaining additional value network parameters. For the same input image, a set of candidate sequences and their corresponding IMS reward values ​​are generated, and the advantage value of each candidate sequence is calculated through within-group statistical analysis to reduce the variance of gradient estimation.

[0061] Specifically, for the i-th candidate LaTeX character sequence, the system calculates its advantage value. The formula is as follows: ; in, The image matching score corresponding to the i-th candidate LaTeX character sequence is... The statistical mean of the matching scores for N candidate LaTeX character sequence images. The statistical standard deviation of the matching scores for N candidate LaTeX character sequence images.

[0062] (5) Construct the loss function and update the network parameters.

[0063] Based on the advantage value data calculated in the preceding steps, a loss function is constructed that includes an alternative objective term and a KL divergence regularization term to guide parameter updates. The alternative objective term uses the importance sampling ratio (i.e., the probability ratio of the current policy to the old policy) and a truncation parameter to limit the policy update magnitude and prevent drastic oscillations in model parameters; the KL divergence regularization term is used to constrain the current policy distribution from deviating too far from the reference policy distribution.

[0064] To minimize the target loss, the loss function defined in this embodiment is... Specifically as follows: ; in, Indicates the importance sampling ratio; This represents the truncation function. For truncation parameters (preferred) ); Denotes KL divergence, KL coefficient (preferred) ).

[0065] Finally, a gradient descent optimizer (such as Adam) is used to backpropagate the loss function, calculate the gradient, and update the policy network parameters. During training, the parameters of the trained policy network are periodically (e.g., every 4 iterations) synchronized with the old policy network and the reference policy network.

[0066] like Figure 2 As shown, the overall framework of the above training process is illustrated. Figure 2 The handwritten mathematical formula images and corresponding LaTeX character sequences shown are for illustrative purposes only. In practical applications, users can input the corresponding handwritten mathematical formula images according to their actual needs.

[0067] III. Effect Verification.

[0068] To verify the effectiveness and universality of this invention, extensive tests were conducted on various public datasets, including CROHME 2014 / 2016 / 2019, HME100K, and M2E, targeting different architecture-based models, including RNN-based ABM and Transformer-based CoMER and PosFormer. Experimental results show that this invention has significant generalization ability and can stably improve the recognition accuracy of various base models: on the CROHME series of standard test sets, the expression recognition accuracy of the PosFormer and CoMER models both achieved an average relative improvement of approximately 1.20%; on the M2E dataset containing complex two-dimensional structures, the recognition accuracy of the PosFormer and CoMER models improved by approximately 1.23% and 1.56%, respectively. These data demonstrate that the image matching-based reward mechanism proposed in this invention can effectively capture subtle structural errors and significantly improve the visual fidelity and structural accuracy of handwritten mathematical formula recognition.

[0069] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0070] Example 2 This invention also provides a high-precision handwritten mathematical formula recognition system, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 3 As shown, the system mainly includes: Network building blocks are used to build policy networks; The network training unit is used to train the policy network using the training set. This includes: performing feature encoding and decoding sampling on handwritten mathematical formula images in the training set based on the policy network to generate several candidate LaTeX character sequences; transforming each candidate LaTeX character sequence and the corresponding ground truth LaTeX character sequence in the training set using a rendering engine to obtain several predicted formula images and one ground truth formula image; calculating the image-level matching degree between each predicted formula image and the ground truth formula image, using the image-level matching degree as the reward value for reinforcement learning, and updating the policy network using a group relative policy optimization method. The recognition unit is used to input the handwritten mathematical formula image to be recognized into the trained policy network after training, obtain candidate LaTeX character sequences, and convert them into predicted formula images through the rendering engine.

[0071] Since the main technical details of each unit in this system have been described in detail in previous embodiments, they will not be repeated here.

[0072] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0073] Example 3 The present invention also provides a processing device, such as Figure 4 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0074] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0075] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example: Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc. The output device can be a display terminal; The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0076] Example 4 The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0077] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0078] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A high-precision method for recognizing handwritten mathematical formulas, characterized in that, include: Construct a policy network; The policy network is trained using the training set, including: performing feature encoding and decoding sampling on handwritten mathematical formula images in the training set based on the policy network to generate several candidate LaTeX character sequences; transforming each candidate LaTeX character sequence and the corresponding ground truth LaTeX character sequence in the training set using a rendering engine to obtain several predicted formula images and one ground truth formula image; calculating the image-level matching degree between each predicted formula image and the ground truth formula image, using the image-level matching degree as the reward value for reinforcement learning, and updating the policy network using a group relative policy optimization method; After training, the handwritten mathematical formula image to be recognized is input into the trained policy network to obtain candidate LaTeX character sequences, which are then converted into predicted formula images by the rendering engine.

2. The high-precision handwritten mathematical formula recognition method according to claim 1, characterized in that, The step of calculating the image-level matching degree between each predicted formula image and the ground value formula image includes: Each prediction formula image and the true value formula image are preprocessed separately. Each preprocessed predicted formula image and the true formula image are converted into a corresponding one-dimensional integer sequence; Calculate the Levinstein edit distance between the one-dimensional integer sequence corresponding to each predicted formula image and the one-dimensional integer sequence corresponding to the true formula image, and normalize it according to the maximum length of the one-dimensional integer sequence to obtain the image matching score, which is used as the image-level matching degree.

3. The high-precision handwritten mathematical formula recognition method according to claim 2, characterized in that... The preprocessing of each predicted formula image and the true formula image includes: The data matrices of each predicted formula image and the true formula image are filled with the same height dimension and the content is vertically centered. Then, the filled data matrices are binarized using a fixed threshold. If the pixel value is greater than the fixed threshold, it is set to 1; otherwise, it is set to 0. Finally, all column vectors with completely zero pixel values ​​in each image data matrix are scanned and removed.

4. The high-precision handwritten mathematical formula recognition method according to claim 2, characterized in that, The formula for calculating the image matching score is: ； in, Let G be the one-dimensional integer sequence corresponding to the truth formula image G. For a single predicted formula image P, there is a one-dimensional integer sequence. A one-dimensional integer sequence and Lewinstein's edit distance; and Represent one-dimensional integer sequences respectively and Length; To predict the image matching score between the formula image P and the true formula image G.

5. The high-precision handwritten mathematical formula recognition method according to claim 1, characterized in that, When training the policy network, initialize the policy network. Simultaneously instantiate the old policy network. and reference policy network Furthermore, during the training of the policy network, the old policy network is periodically updated based on the parameters of the updated policy network. and reference policy network The parameters; the feature encoding and decoding sampling of handwritten mathematical formula images in the training set based on the policy network to generate several candidate LaTeX character sequences refers to using the old policy network that is periodically updated based on the policy network. Feature encoding and decoding sampling are performed on the handwritten mathematical formula images in the training set to generate several candidate LaTeX character sequences; The policy network is updated as follows: based on image-level matching degree, a group relative policy optimization method is used to calculate the advantage value corresponding to each candidate LaTeX character sequence; and, based on the policy network... With the old policy network Calculate the importance sampling ratio; combine the importance sampling ratio, dominance value, and cutoff parameter to calculate the alternative target term; based on the policy network. With reference policy network Calculate the KL divergence regularization term; construct a loss function by combining the alternative objective term and the KL divergence regularization term, and update the policy network through the loss function.

6. The high-precision handwritten mathematical formula recognition method according to claim 5, characterized in that, The calculation of the advantage value for each candidate LaTeX character sequence based on image-level matching degree and using a group relative strategy optimization method includes: For a single handwritten mathematical formula image, N candidate LaTeX character sequences are generated; for the i-th candidate LaTeX character sequence, the image-level matching degree between its corresponding predicted formula image and the true formula image is denoted as . Advantage value Calculated using the following formula: ； in, The statistical mean of the image-level matching degree of N candidate LaTeX character sequences. Let be the statistical standard deviation of the image-level matching degree of N candidate LaTeX character sequences.

7. A high-precision handwritten mathematical formula recognition method according to claim 5 or 6, characterized in that, The loss function is expressed as: ； in, Let be the loss function, and be the mathematical expectation. To replace the target item, For KL divergence regularization, Indicates the importance sampling ratio, Represents a sequence of historical LaTeX characters. Generate current tag The conditional state, where t is the index in the LaTeX character sequence, t=1,…T, and T is the length of the LaTeX character sequence. and Representing policy networks With the old policy network Based on historical LaTeX character sequences Generate current tag The probability distribution; This represents the truncation function. To truncate parameters; Denotes KL divergence, This is the KL coefficient.

8. A high-precision handwritten mathematical formula recognition system, characterized in that, To implement the method according to any one of claims 1 to 7, comprising: Network building blocks are used to build policy networks; The network training unit is used to train the policy network using the training set. This includes: performing feature encoding and decoding sampling on handwritten mathematical formula images in the training set based on the policy network to generate several candidate LaTeX character sequences; transforming each candidate LaTeX character sequence and the corresponding ground truth LaTeX character sequence in the training set using a rendering engine to obtain several predicted formula images and one ground truth formula image; calculating the image-level matching degree between each predicted formula image and the ground truth formula image, using the image-level matching degree as the reward value for reinforcement learning, and updating the policy network using a group relative policy optimization method. The recognition unit is used to input the handwritten mathematical formula image to be recognized into the trained policy network after training, obtain candidate LaTeX character sequences, and convert them into predicted formula images through the rendering engine.

9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images