Sequence-based encoder-decoder approach for handwritten mathematical expression recognition

Abstract

The application discloses a handwriting mathematical expression recognition method based on a sequence-based encoder-decoder, and belongs to the technical field of document analysis and pattern recognition, and comprises the following steps: obtaining a handwriting mathematical expression image and performing pretreatment to obtain an input image; a DenseNet backbone network is used to extract a high-dimensional visual feature map of the input image; a counting-based perception position module is constructed, including a counting symbol task and a position learning task; training and reasoning are performed, expression recognition loss, position loss and counting loss are jointly optimized in the training stage, and only the counting symbol task is enabled in the reasoning stage; and handwriting mathematical expression recognition is performed based on the trained model. According to the application, no additional symbol-level labeling is needed, the symbol feature learning ability and the spatial position perception ability can be improved at the same time, the recognition precision is significantly improved, the application has strong robustness to complex structures and similar symbols, and no additional calculation overhead is generated in the reasoning stage.

Description

Technical Field

[0001] This invention relates to the field of document analysis and pattern recognition technology, and in particular to a sequence-based encoder-decoder method for recognizing handwritten mathematical expressions. Background Technology

[0002] Handwritten Mathematical Expression Recognition (HMER) is a core task in the field of document analysis and pattern recognition, aiming to convert images of handwritten mathematical expressions into structured LaTeX sequences. However, due to the complex hierarchical structure of mathematical expressions (such as multi-level fractions, nested radicals, large matrices, subscripts and superscripts, braces, etc.), visual similarities between symbols (such as the Latin letter 'a' and the Greek letter 'α', 'o' and '0'), and the high diversity of handwriting styles (scale, stroke differences), existing HMER methods still face significant challenges.

[0003] Mainstream methods are primarily divided into two categories: tree-based serialization methods and sequence-based encoder-decoder methods. While the former (such as BLSTM, TSDNet, and SAN) can model hierarchical relationships well, they rely on complex structural annotations and strict grammatical rules, resulting in problems such as error propagation and poor generalization ability. The latter (such as WAP, DenseWAP, CoMER, PosFormer, and CAN) treats HMER as an image-to-sequence transformation, using attention mechanisms to directly predict LaTeX. However, they still commonly suffer from "over-parsing" (generating redundant structural symbols) and "under-parsing" (omitting necessary structural or content symbols), especially when processing structural symbols (such as ˆ, _, \sqrt, \frac), making it difficult to accurately align visual features with hierarchical spatial relationships.

[0004] Patent application CN2025101127027 discloses a handwritten mathematical expression recognition and comparison method based on large model inference, which can reduce syntax errors in predicting LaTeX character sequences and ambiguity in the text to be recognized, but its recognition accuracy for structural symbols is insufficient and its ability to learn symbol features is poor. Patent application CN2025101127046 discloses a handwritten mathematical expression recognition method based on context-consistent Bayesian priors, which can effectively improve the recognition accuracy of handwritten mathematical expressions, but its ability to learn symbol features is poor. Patent application CN2023100868494 discloses a handwritten mathematical expression recognition method based on octave convolution and encoding / decoding, which has good generalization ability but insufficient recognition accuracy. Patent application CN2019108968085 discloses a deep learning-based method for recognizing running race bib numbers, which can better parse the two-dimensional structure in mathematical expressions and extract image features, but its recognition accuracy for symbols is insufficient and its ability to recognize special structural symbols is poor.

[0005] Encoder-decoder architectures have been widely used in recent HMER methods, which formulate HMER as an image-to-sequence transformation problem. Given a handwritten formula, an attention mechanism is used to predict its corresponding symbol sequence (e.g., LaTeX). This treats the mathematical expression as a LaTeX sequence and employs an encoder-decoder architecture to predict each symbol in an autoregressive manner. However, due to the complex hierarchical structure of mathematical expressions, visual similarities between symbols, and the high diversity of handwriting styles (scale, stroke differences), existing HMER methods are still prone to over- and under-parsing when handling this task. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention proposes a sequence-based encoder-decoder method for recognizing handwritten mathematical expressions. This method can simultaneously enhance symbolic feature learning and spatial location awareness without additional symbol-level annotation, thereby significantly improving the recognition accuracy of handwritten mathematical expressions, especially its robustness to complex expressions containing similar symbols, Greek letters, special symbols, and deeply nested structures.

[0007] This application discloses a sequence-based encoder-decoder method for recognizing handwritten mathematical expressions, comprising the following steps: S1. Obtain the image of the handwritten mathematical expression and perform preprocessing to obtain the input image; S2. Use the DenseNet backbone network to extract high-dimensional visual feature maps of the input image; S3. Construct a counting-based position perception module, which includes a counting symbol task and a position learning task, and achieves collaborative perception through an interactive mechanism that guides position prediction using counting vectors. S4. Training and Inference: During the training phase, the expression recognition loss, location loss, and counting loss are jointly optimized. During the inference phase, only the counting symbol task is enabled to complete LaTeX sequence prediction. S5. Handwritten mathematical expression recognition based on training from S4.

[0008] Preferably, S1 includes: Extracting the corresponding LaTeX sequence from the InkML format of the handwritten ink marks corresponding to the image of the handwritten mathematical expression. , LaTeX sequence groundtruth1 serves as the labeled data during training.

[0009] Preferably, the preprocessing includes random rotation, scaling, translation, and stroke perturbation.

[0010] Preferably, the counting symbol task in S3 includes: High-dimensional visual feature maps Two parallel convolutional branches are used to obtain fused features. ; Fusion features Weighted features are obtained through channel attention processing. ; Based on weighted features Generate multi-channel counting plot ; Multi-channel counting graph The cross-inhibition mechanism was used to obtain the result. ; right Global Sum Pooling is performed on each channel to obtain a counting vector. .

[0011] Preferably, the weighted features The calculation formula is as follows:

[0012] in, For channel attention, , These are the weights for the ReLU function.

[0013] Preferably, the cross-suppression mechanism includes: right Each channel Calculate the average activation value of other channels as the inhibition factor:

[0014] and utilize Suppression treatment is performed, among which It is an adjustable hyperparameter.

[0015] Preferably, the location learning task in S3 includes: Recursively parse substructures from the target sequence according to LaTeX syntax rules to construct a position tree; Generate a sequence of tree path identifiers for each symbol as a location identifier; The location identifier is padded and embedded to obtain a location embedding vector. ; High-dimensional visual feature maps With weighted features The fusion yields a vector and embed the position into the vector. with vector The input is a three-layer decoder based on Transformer; Counting vector Expanded into a guidance matrix The guidance matrix After multiplying by the decoder attention score and passing through the softmax function, we obtain... This enables position prediction and error correction under counting constraints.

[0016] Preferably, the Transformer-based three-layer decoder includes multi-head attention, an attention refinement module, and a feedforward network.

[0017] Preferably, the total loss function for the training phase described in S4 is as follows:

[0018] Total mission losses LaTeX sequence Standard cross-entropy and positional loss compared to the results of recognizing LaTeX expressions. The counting loss is the sum of the cross-entropy of the labeled data ground truth2 (which has the ability to acquire location information from mathematical expressions) and the prediction results of relative spatial locations. SmoothL1 regression was used for calculation. Loss compared to groundtruth1.

[0019] Preferably, the reasoning phase in S4 includes: Only run the counting symbol CS task to generate the counting vector Extended into a guidance matrix via MLP ; Compare the high-dimensional visual feature map V′ with the guidance matrix Inputting a three-layer Transformer-based decoder, the system generates LaTeX sequences via autoregression. .

[0020] The beneficial effects of this invention are: (1) Compared with the benchmark model, the expression recognition rate of the present invention is significantly improved, and the improvement is significant under the fault tolerance index, which fully verifies the advantages of the present invention in the recognition of complex structures and similar symbols.

[0021] (2) This invention reduces confusion in similar symbol counting maps by using the cross-inhibition mechanism of the counting symbol task and combines the dynamic position tree encoding of the position learning task to enable the model to accurately align visual features with hierarchical spatial relationships, thereby fundamentally alleviating the phenomena of over-analysis and under-analysis.

[0022] (3) The combination of multi-scale convolution, channel attention and cross suppression in this invention enables the model to learn the subtle differences in similar symbols more effectively, while significantly enhancing the robustness of the recognition of Greek letters and special symbols.

[0023] (4) The present invention is based on a location tree recursively constructed using LaTeX syntax, which can assign a unique tree path identifier to each symbol. Combined with a counting-guided attention mechanism, the model has a stronger spatial awareness when dealing with complex structures such as multi-level fractions and nested radicals.

[0024] (5) In the inference stage of this invention, only the counting symbol task is enabled, and the position learning task is not required. This does not increase the additional computational overhead, and the inference speed is consistent with the benchmark model, which has good practical deployment value. Attached Figure Description

[0025] Figure 1 This is a flowchart of a sequence-based encoder-decoder method for recognizing handwritten mathematical expressions according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the location tree encoding according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the training phase in an embodiment of the present invention; Figure 4 This is a schematic diagram of the reasoning stage in an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided with reference to the accompanying drawings and embodiments.

[0027] This application discloses a sequence-based encoder-decoder method for recognizing handwritten mathematical expressions, the process of which is as follows: Figure 1 As shown, it includes the following steps: S1. Preparation Stage: Data Preprocessing. Obtain the image of the handwritten mathematical expression and perform preprocessing to obtain the input image.

[0028] In this embodiment, the handwritten mathematical expression images are taken from the publicly available benchmark dataset CROHME series (CROHME2014 / 2016 / 2019 test sets).

[0029] Extracting the corresponding LaTeX sequence from the handwritten ink in InkML format corresponding to the image of the handwritten mathematical expression:

[0030] in, Indicates length is A sequence of mathematical expressions in LaTeX form.

[0031] LaTeX sequences Ground truth1 serves as the labeled data during training.

[0032] Data augmentation processing is performed on the image of the handwritten mathematical expression to obtain the input image. Optionally, data augmentation processes include random rotation, scaling, translation, stroke perturbation, etc., to improve generalization.

[0033] S2, Encoding stage: Construct the DenseNet backbone network and extract visual features.

[0034] With the preprocessed input image As input, and These represent the height and width of the image, respectively. The publicly available DenseNet feature extraction network architecture is used as the backbone network to output a high-dimensional visual feature map. It is used to capture stroke textures, symbol shapes, and local layouts.

[0035] S3. Construct a counting-based position-aware module (CP module).

[0036] The CP module mainly consists of two tasks: counting symbols (CS) and position learning (PL). The two achieve collaborative sensing through an interactive mechanism that guides position prediction using counting vectors.

[0037] Symbol counting (CS) task. To address differences in writing scale / stroke, a multi-scale approach with cross-suppression is employed, using two parallel convolutional branches.

[0038] Input high-dimensional visual feature map Input the following two convolution branches respectively: First branch: First, a 3×3 convolution is performed (where the padding edge padding size is set to 1 and the number of groups in the convolution is set to 3), then batch normalization is performed using BatchNorm, and finally the ReLU nonlinear activation function is input.

[0039] The second branch first uses a 5×5 convolution (with padding set to 2 and groups set to 32), then performs batch normalization using BatchNorm, and finally inputs the ReLU nonlinear activation function.

[0040] The two branches' output features are fused with equal weights to obtain , Input SE-like channel attention via the ReLU function (reduction=16) yields the weighted features :

[0041] in, , These are the weights for the ReLU function.

[0042] The dimensionality is reduced to D=111 channels through a 1×1 convolution operation, then activated by the Sigmoid function to generate values ​​in the range (0,1), and then mapped to obtain a multi-channel counting map. ,in , , These represent the height, width, and dimensions of the multi-channel counting map.

[0043] Simultaneously utilizing the cross-inhibition mechanism, Each channel Calculate the average activation value of other channels as the inhibition factor:

[0044] and utilize Suppression treatment is performed. It is an adjustable hyperparameter.

[0045] This mechanism reduces confusion in count maps of similar symbols (such as o and 0). Global SumPooling feature pooling is performed on each channel to obtain a counting vector. , This represents the number of occurrences of 111 symbols, including "[sos]", "[eos]", and "[pad]", in the LaTeX form of the mathematical expression.

[0046] Position learning (PL) task. Based on LaTeX syntax rules, recursively parse substructures (superscripts / subscripts, \frac, \sqrt, etc.) from the target sequence to construct a position tree: The main symbol is the Root node, the superscript / upper part is the Child node, and the subscript / lower part is the Parent node. The sibling substructures are sorted according to their relative left and right positions in the image (or horizontal center coordinates).

[0047] Generate a sequence of tree path identifiers for each symbol as a location identifier. For example: root node: R; root's child: RC; root's parent's child: RPC.

[0048] For example, such as Figure 2 As shown. First, the LaTeX sequence is divided into five substructures: "+", " "+" and " Secondly, different symbols within a substructure will generate different branches, forming a relative position tree. For example, in the first substructure " In the middle, the main body The exponent "8" and the subscript "2" belong to the upper and lower parts, respectively encoded into the child node "Child" and the parent node "Parent" of the corresponding tree. Once all substructures have been encoded into the tree, they are combined into a positional tree structure.

[0049] In summary, using the above rules, each symbol is assigned a position identifier in the tree. , to indicate its relative spatial position.

[0050] All location identifiers are padded with whitespace symbols to the maximum length. , get location ( (where is the sequence length), serving as groundtruth2, a labeled data used to obtain positional information from mathematical expressions.

[0051] Furthermore, considering that these identifiers have different lengths, they are padded to a uniform length using the "[pad]" marker. A "[sos]" marker is added before each identifier to indicate its start and end. The processed identifiers are then organized into a matrix. ,matrix Through a nonlinear embedding layer: linear projection .

[0052] Superimposed learnable absolute position coding The final position embedding vector is obtained. .

[0053] Construct a three-layer decoder block based on Transformer. Then, embed the vector... Together with high-dimensional visual feature maps With weighted features fused vector The data is fed into a three-layer Transformer-based decoder block. This block mainly consists of a public module (MHA), an attention refinement module (ARM), and a feedforward network (FFN). Simultaneously, a counting-guided attention MLP is introduced into the decoder, with the counting vector... Extended into a guidance matrix via MLP Guidance Matrix After multiplying by the decoder attention score and passing through the softmax function, we obtain... To achieve position prediction error correction under counting constraints, a transformer-based decoding process is incorporated. Finally, these inputs are processed through a three-layer decoder block based on the transformer. , To generate output features It is used to predict nesting levels and relative positions.

[0054] S4, Training and Reasoning.

[0055] Training phase, such as Figure 3 As shown, it includes: Using the publicly available benchmark dataset CROHME, the input image is first extracted through the DenseNet backbone network. High-dimensional visual feature map .

[0056] Subsequently, the high-dimensional visual feature map The input is fed into the CP module to predict the number of symbols in each class and generate a one-dimensional count vector representing the counting results. .

[0057] Based on this, position encoding is performed. After position encoding is completed, the position is obtained. , to indicate relative spatial position.

[0058] In the decoder, a counting-guided attention vector S is introduced and expanded into a guidance matrix via an MLP. Guidance Matrix After multiplying by the decoder attention score and passing through the softmax function, we obtain... This enables position prediction and error correction under counting constraints.

[0059] embed vector Together with high-dimensional visual feature maps With weighted features fused vector The input is fed into a Transformer-based three-layer decoder block to learn its symbolic features, and then a parallel linear head is deployed to recognize LaTeX expressions. Combined with expression recognition, joint optimization is performed on the position-aware task to facilitate the learning of position-aware symbolic features.

[0060] Specifically, the main purpose of this process is to transform the mathematical expression of the sequence into positions. and counting vector This allows us to obtain its location and symbolic feature information. Using this information, parallel recognition heads are employed to parse the number of symbol classes and location information within the modeled location tree.

[0061] The total loss function is as follows:

[0062] Total mission losses LaTeX sequence Standard cross-entropy and positional loss compared to the results of recognizing LaTeX expressions. The sum of the cross-entropy between ground truth2 and the predicted relative spatial location, and the counting loss. SmoothL1 regression was used for calculation. The loss of ground truth1.

[0063] Reasoning stage, such as Figure 4 As shown, only the counting symbol (CS) is performed to generate the counting vector. Extended into a guidance matrix via MLP High-dimensional visual feature maps LaTeX sequence prediction is completed by directly inputting the decoder, without the need for position learning (PL), and the inference speed is consistent with the baseline model.

[0064] S5. Use the DenseNet backbone network trained in S4, the CP module, and the Transformer-based three-layer decoder block to recognize handwritten mathematical expressions.

[0065] Specifically, the input is any handwritten mathematical expression image (photographed, scanned, or handwritten on a tablet).

[0066] Image is obtained through image preprocessing (data augmentation), and high-dimensional visual feature maps are extracted using the DenseNet backbone network. Only the counting symbol (CS) operates in the CP module, generating the counting vector. And expanded into a guidance matrix via MLP Input the high-dimensional visual feature map into the transformer decoder. Direct input decoder Transformer autoregressive generation of LaTeX sequences Structured LaTeX code can be directly rendered as editable formulas.

[0067] The proposed scheme in this embodiment does not require position learning (PL) during inference, has no additional computational overhead, and is comparable in speed to the benchmark model. It also exhibits significant robustness to complex nesting and similar symbols.

[0068] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A sequence-based encoder-decoder approach for handwritten mathematical expression recognition, characterized in that, Includes the following steps: S1. Obtain the image of the handwritten mathematical expression and perform preprocessing to obtain the input image; S2. Use the DenseNet backbone network to extract high-dimensional visual feature maps of the input image; S3. Construct a counting-based position perception module, which includes a counting symbol task and a position learning task, and achieves collaborative perception through an interactive mechanism that guides position prediction using counting vectors. S4. Training and Inference: During the training phase, the expression recognition loss, location loss, and counting loss are jointly optimized. During the inference phase, only the counting symbol task is enabled to complete LaTeX sequence prediction. S5. Handwritten mathematical expression recognition based on training from S4.

2. The sequence-based encoder-decoder based method of handwritten mathematical expression recognition according to claim 1, characterized in that, S1 includes: extracting a corresponding LaTeX sequence from a handwritten ink InkML format corresponding to a handwritten mathematical expression image , the LaTeX sequence as a groundtruth1 at the time of training 3.The sequence-based encoder-decoder based handwritten mathematical expression recognition method of claim 2, wherein, The preprocessing includes random rotation, scaling, translation, and stroke perturbation.

4. The sequence-based encoder-decoder based method of handwritten mathematical expression recognition according to claim 3, characterized in that, The counting symbol task described in S3 includes: High-dimensional visual feature maps Two parallel convolutional branches are used to obtain fused features. ; Fusion features Weighted features are obtained through channel attention processing. ; Based on weighted features Generate multi-channel counting plot ; Multi-channel counting graph The cross-inhibition mechanism was used to obtain the result. ; right Global Sum Pooling is performed on each channel to obtain a counting vector. .

5. The handwritten mathematical expression recognition method based on sequence encoder-decoder according to claim 4, characterized in that, The weighted features The calculation formula is as follows: in, For channel attention, , These are the weights for the ReLU function.

6. The handwritten mathematical expression recognition method based on sequence encoder-decoder according to claim 5, characterized in that, The cross-inhibition mechanism includes the following processing: right Each channel Calculate the average activation value of other channels as the inhibition factor: and utilize Suppression treatment is performed, among which It is an adjustable hyperparameter.

7. The handwritten mathematical expression recognition method based on sequence encoder-decoder according to claim 6, characterized in that, The location learning task described in S3 includes: Recursively parse substructures from the target sequence according to LaTeX syntax rules to construct a position tree; Generate a sequence of tree path identifiers for each symbol as a location identifier; The location identifier is padded and embedded to obtain a location embedding vector. ; High-dimensional visual feature maps With weighted features The fusion yields a vector and embed the position vector with vector The input is a three-layer decoder based on Transformer; Counting vector Expanded into a guidance matrix The guidance matrix After multiplying by the decoder attention score and passing through the softmax function, we obtain... This enables position prediction and error correction under counting constraints.

8. The handwritten mathematical expression recognition method based on sequence encoder-decoder according to claim 7, characterized in that, The Transformer-based three-layer decoder includes multi-head attention, attention refinement module, and feedforward network.

9. The handwritten mathematical expression recognition method based on sequence encoder-decoder according to claim 8, characterized in that, The total loss function for the training phase described in S4 is as follows: Total mission losses LaTeX sequence Standard cross-entropy and positional loss compared to the results of recognizing LaTeX expressions. The counting loss is the sum of the cross-entropy of the labeled data ground truth2 (which has the ability to acquire location information from mathematical expressions) and the prediction results of relative spatial locations. SmoothL1 regression was used for calculation. The loss of ground truth1.

10. The handwritten mathematical expression recognition method based on sequence encoder-decoder according to claim 9, characterized in that, The reasoning phase described in S4 includes: Only run the counting symbol CS task to generate the counting vector Extended into a guidance matrix via MLP ; Compare the high-dimensional visual feature map V′ with the guidance matrix Inputting a three-layer Transformer-based decoder, the system generates LaTeX sequences via autoregression. .

Images