Handwritten mathematical formula identification method based on rotation position coding

Abstract

The invention discloses a handwritten mathematical formula identification method based on rotation position coding, and relates to the technical field of handwritten mathematical formula identification. The method is characterized in that aiming at the problem of insufficient long-range dependence capture in long formula sequence modeling in the existing method, an end-to-end encoder-decoder framework is constructed, a rotation position encoding mechanism is introduced into a self-attention layer of a Transform decoder, and rotation transformation related to the position is applied to a query vector and a key vector, so that the long-range dependence capture in the long formula sequence modeling is realized. Enabling attention to calculate relative position information between explicit modeling symbols; and meanwhile, the attention distribution in the decoding process is constrained in combination with a coverage attention mechanism, so that repeated attention and omission of symbols are reduced. The method effectively solves the problem that long-range dependence capture is insufficient in handwritten mathematical formula long sequence modeling in an existing method, improves the recognition accuracy and robustness, and is suitable for application scenes such as mathematical formula input, intelligent education and document digitization.

Description

Technical Field

[0001] This invention belongs to the field of handwritten mathematical formula recognition technology, specifically relating to a handwritten mathematical formula recognition method based on rotational position encoding. Technical Background

[0002] Handwritten mathematical formula recognition is a key enabling technology for realizing intelligent education, human-computer interaction, and the digitization of scientific literature, and has broad application prospects.

[0003] Before the rise of deep learning, traditional methods mostly followed a pipeline of "symbol segmentation → symbol classification → structural analysis". These methods heavily rely on the accuracy of the initial segmentation. When faced with complex handwriting scenarios such as overlapping symbols and intersecting strokes, errors accumulate and amplify along the process, leading to system failure.

[0004] Subsequently, with the development of deep learning, especially the popularization of sequence-to-sequence models, researchers began to abandon explicit segmentation and turn to an end-to-end learning paradigm: directly mapping input images to LaTeX sequences, significantly improving system robustness and overall performance. Existing end-to-end handwritten mathematical formula recognition methods mainly include the following two technical solutions:

[0005] 1. CNN-RNN-based architecture: A convolutional neural network is used as the encoder to extract visual features, and a recurrent neural network is used as the decoder to autoregressively generate sequences. Although an attention mechanism is introduced to dynamically align image regions, the inherent sequential computation mode and weak long-range dependency modeling ability of RNNs limit their performance in processing complex and lengthy formulas.

[0006] 2. CNN-Transformer-based architecture: The Transformer, with its fully attention-based parallel computing architecture, has completely replaced RNNs as the standard decoder. Its powerful global modeling capabilities theoretically allow it to better capture long-range dependencies between output symbols. The CoMER model successfully introduced a coverage attention refinement mechanism into the Transformer decoder, effectively mitigating the attention dispersion problem by utilizing historical alignment information to refine current attention weights while maintaining the model's parallel training efficiency. Building upon CoMER, subsequent research has primarily focused on adding auxiliary tasks such as counting awareness and tree structure prediction to further improve performance.

[0007] However, the aforementioned techniques based on attention optimization and task enhancement still suffer from a bottleneck that remains largely unresolved: the underlying positional encoding mechanism in the decoder used to model sequence order and dependencies is inadequate. Both CoMER and subsequent improved models typically employ only traditional absolute positional encoding at the bottom layer of their Transformer decoders. When sequences are long, these methods struggle to effectively maintain the relative positional relationships between distant symbols during attention computation, resulting in limited ability to model deeply nested mathematical structures and long-range logical dependencies.

[0008] Therefore, fundamentally improving the positional encoding mechanism of the decoder to enhance the model's ability to perceive and understand complex two-dimensional relationships and long-range dependencies in long formulas has become a key challenge in improving the performance of handwritten mathematical formula recognition. Rotational Positional Encoding (RoPE) has been proven effective in improving long sequence modeling capabilities in natural language processing. By rotating the query and key vectors in the attention mechanism, it allows the vector inner product to directly reflect the relative positional relationships of sequence elements, providing a new approach to solving long-range dependency problems in sequence modeling tasks. Summary of the Invention

[0009] Against this backdrop, the present invention aims to address the problem of insufficient long-range dependency modeling capability in existing handwritten mathematical formula recognition methods due to limitations in the basic positional encoding mechanism of the decoder. However, because handwritten mathematical formulas possess a significant two-dimensional spatial structure and long-range symbolic dependencies, their symbol sequence order differs fundamentally from that of natural language sequences. Furthermore, complex two-dimensional nested relationships exist between symbols, such as subscripts, superscripts, fractions, and square roots. Therefore, although rotational positional encoding has been applied in the field of natural language processing, existing technologies have not incorporated rotational positional encoding into the self-attention modeling of handwritten mathematical formula recognition decoders.

[0010] To achieve the above objectives, this invention provides a handwritten mathematical formula recognition method based on rotational position encoding. The core of this invention lies in introducing rotational position encoding into the self-attention layer of the Transformer decoder. By directly applying rotational transformations to the query vector and key vector, the attention score can reflect the relative positional information between symbols. This allows the model to model dependencies between symbols at arbitrary distances without introducing additional learnable parameters, thus improving the recognition ability for long formulas and nested structures.

[0011] The technical solution adopted in this invention is:

[0012] A handwritten mathematical formula recognition method based on rotational position encoding includes the following steps:

[0013] Step 1: Obtain the image of the handwritten mathematical formula and preprocess the image of the handwritten mathematical formula;

[0014] Step 2: Construct a handwritten mathematical formula recognition model, which includes a visual encoder and a Transformer decoder integrating rotational position encoding; the rotational position encoding is used to perform position-related rotational transformations on the query vector and key vector in the self-attention calculation of the decoder.

[0015] Step 3: Construct a loss function and train the handwritten mathematical formula recognition model;

[0016] Step 4: Input the image of the handwritten mathematical formula to be recognized into the trained handwritten mathematical formula recognition model, and output the corresponding LaTeX symbol sequence.

[0017] Furthermore, the specific process of step 1 is as follows:

[0018] The original image of the handwritten mathematical formula is obtained. First, to enhance the model's robustness to input scale changes, the image undergoes random scaling during the training phase. Then, the image is size-normalized, constraining its height and width within a preset effective range to adapt to the model's input requirements. Finally, the image is padded and masked to generate a uniform batch tensor.

[0019] Furthermore, the specific process of step 2 is as follows:

[0020] The visual encoder employs a deep convolutional neural network as its backbone. Its function is to perform multi-level visual feature extraction on preprocessed standardized handwritten formula images. The encoder receives the input image and then performs convolution and pooling operations sequentially on the preprocessed handwritten mathematical formula image for preliminary feature extraction. The extracted features are then fed into multiple dense feature extraction modules and a transition module. The dense feature extraction module consists of multiple ordinary and deformable convolutional layers, connected by feature cascading. The transition module performs channel compression and spatial downsampling. After passing through the three dense feature extraction modules and corresponding transition modules, a feature map is output. Where H, W, and C represent the height, width, and number of channels of the feature map, respectively. This feature map is flattened and embedded into a visual feature sequence. ,in Let L be the sequence length and D be the feature dimension. Then, a two-dimensional positional encoding is introduced to represent the positional information of the features in two-dimensional space. Finally, the visual feature sequence with the added two-dimensional positional encoding is used as the key and value input for the cross-attention of the decoder.

[0021] The decoder is an improved decoder based on the Transformer architecture, consisting of multiple identical decoding layers stacked together. Each decoding layer includes a multi-head self-attention module integrating rotational position encoding, a cross-attention module integrating overlay attention refinement mechanism, and a feedforward neural network. The multi-head self-attention module models the contextual dependencies within the generated symbol sequence; the cross-attention module aligns the decoder state with the visual feature sequence output by the visual encoder at the current decoding time step, guiding the symbol generation process to focus on the image region associated with the current symbol.

[0022] In the multi-head self-attention module, before calculating the self-attention score, the query vector at position m is... and the key vector at position n A rotational position encoding transformation is applied while preserving the value vector. This transformation first defines a position-dependent angular frequency sequence. Pre-calculation of cosine tensor Sine tensor ,satisfy , Then, the vector is divided into several two-dimensional sub-vectors along the feature dimension according to adjacent pairs of elements, and an orthogonal rotation transformation is calculated for each sub-vector, i.e. Record this operation as .

[0023] Multiply the original vector and the rotated vector element-wise with the cosine tensor and the sine tensor respectively, and then add them together to obtain the rotated representation:

[0024]

[0025] Where ⊙ represents element-wise multiplication.

[0026] The transformed query and key vector are used for standard dot product attention calculation in the self-attention module. This inner product can be expressed as:

[0027]

[0028] in It is a two-dimensional rotation matrix. It is an angular frequency sequence, with the corresponding frequency applied to each dimension pair. .

[0029] Due to the orthogonality of the rotation matrix, the inner product result depends only on the relative position (nm) rather than the absolute position, thus enabling explicit modeling of the relative positional relationships between arbitrary distance symbols without introducing additional learnable parameters, enhancing the model's ability to understand long formulas and complex nested structures.

[0030] The cross-attention module, which integrates a coverage attention refinement mechanism, is used to refine the attention distribution of visual features during the decoding process. At each decoding time step t, the cross-attention module first calculates the cross-attention score and its attention weight for the current time step based on the current decoding state and the visual encoder output features. To fully utilize the attention information generated during historical decoding, the coverage attention refinement mechanism introduces cumulative modeling of the cross-attention weights from historical time steps. Specifically, for historical time steps... Cross attention weights Introduce a time-weighted coefficient related to the current time step t. The time weighting coefficient is determined based on the time interval (t−i) between the current time step and the historical time step, and is used to differentiate and weight the attention weights of different historical time steps.

[0031] Based on this, the historical cross-attention weights after time weighting are accumulated to obtain the coverage vector corresponding to the current time step. At each step, the covering vector passes through a convolutional layer, an activation function, a linear layer, and a pooling operation in sequence, resulting in a covering penalty. This is combined with the original cross-attention score e to correct the cross-attention distribution at the current time step, suppressing repeated attention to already sufficiently focused regions. The formula is expressed as follows: ,in, The attention score is the result of suppression.

[0032] The decoder output is finally passed through a linear projection layer and a Softmax activation function, mapped onto the probability distribution of the target character set, and the final LaTeX sequence is generated in an autoregressive manner.

[0033] Furthermore, the specific process of step 3 is as follows:

[0034] A cross-entropy loss function is constructed as the main optimization objective for model training; preprocessed handwritten mathematical formula images and their corresponding LaTeX sequence labels are used as training data; the handwritten mathematical formula recognition model is trained in an end-to-end manner, and the parameters of the visual encoder and the decoder integrating rotation position encoding are iteratively optimized through the backpropagation algorithm, so that the model learns to accurately map from image features to symbol sequences.

[0035] Furthermore, the specific process of step 4 is as follows:

[0036] During the inference phase, the pre-processed image to be identified is input into the trained handwritten mathematical formula recognition model. The model's visual encoder extracts image features, and the decoder, which integrates rotational position encoding, generates the LaTeX symbol sequence with the highest probability autoregressively through the relative position modeling mechanism introduced in its self-attention module during the decoding process, as the final recognition result output. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the handwritten mathematical formula recognition method based on rotational position encoding according to the present invention. Detailed Implementation

[0038] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0039] This invention proposes a handwritten mathematical formula recognition method based on rotational position encoding. By introducing a rotational position encoding mechanism into the self-attention layer of the Transformer decoder, a position-related rotational transformation is applied to the query and key vectors, making the attention score explicitly dependent on the relative position between symbols. This directly encodes the relative position information into the attention mechanism, enhancing the model's ability to model long formulas and complex nested structures (such as fractions, superscripts and subscripts).

[0040] This method employs an end-to-end encoder-decoder architecture: the visual encoder extracts multi-level features from handwritten formula images based on a deep convolutional neural network; the decoder uses a Transformer structure with integrated rotation position coding to generate LaTeX symbol sequences in an autoregressive manner.

[0041] like Figure 1 As shown, a handwritten mathematical formula recognition method based on rotational position encoding specifically includes the following steps:

[0042] Step 1: Preprocess the image of the handwritten mathematical formula. The specific process is as follows:

[0043] Step 1.1: Obtain image data of CROHME's handwritten mathematical formulas and create a dataset.

[0044] Step 1.2: Perform a random scaling transformation on the original image. This step is performed only during the training phase and is used for data augmentation to improve the model's robustness to changes in input scale. Specifically, the scaling ratio is set to a range of 0.7 to 1.4, and a scaling factor is randomly and uniformly sampled within this range. Bilinear interpolation is then used to scale both the height and width of the image simultaneously to obtain the scaled image.

[0045] Step 1.3: Perform image size normalization. This step ensures that the size of all images is within a preset effective range, avoiding images that are too large or too small from affecting feature extraction results. Specifically, set the effective range for image height (16 to 256 pixels in this embodiment) and the effective range for width (16 to 1024 pixels in this embodiment). If the image exceeds the effective range, bilinear interpolation is used to proportionally enlarge or reduce the image size to ensure it remains within the effective range.

[0046] Step 1.4: Image padding and masking. To enable parallel processing of images of different sizes in batches, all images within a batch need to be padded to the same size. Specifically, using the largest image size in the batch as a baseline, other images are padded with zero values ​​on the right and bottom. Simultaneously, a corresponding mask matrix is ​​generated to identify the valid and padded areas in the image, so that the influence of the padded portion can be ignored in subsequent processing.

[0047] Step 2: Construct a handwritten mathematical formula recognition model. The specific process is as follows:

[0048] The handwritten mathematical formula recognition model consists of two main parts: a visual encoder and a decoder that integrates rotational position encoding.

[0049] Step 2.1, Construction of the visual encoder.

[0050] The visual encoder employs a deep convolutional neural network as its feature extraction backbone to extract high-level semantic visual features from preprocessed images of handwritten mathematical formulas. In a preferred embodiment, the visual encoder is implemented using a DenseNet convolutional neural network with a densely connected structure, and the input image size is [insert size here]. , where b represents the batch size.

[0051] First, the input image is passed through an initial convolutional layer and a pooling layer for preliminary feature extraction. The initial convolutional layer uses a 7×7 convolutional kernel with a stride of 2. Then, the pooling operation reduces the feature map space size to about 1 / 4 of the original image.

[0052] After initial feature extraction, the feature maps are sequentially input into multiple sets of dense feature extraction modules and transition modules. Each dense feature extraction module consists of multiple convolutional layers, including ordinary convolutional layers and deformable convolutional layers. The convolutional layers are connected using a feature cascading method, allowing the features output from the previous layer to be used as input to the next layer, thereby achieving feature reuse and multi-scale information fusion. The transition modules are positioned between adjacent dense feature extraction modules to perform channel compression and spatial downsampling operations, controlling the model parameter scale and gradually reducing the spatial resolution of the feature maps. In a preferred embodiment, the transition module includes a 1×1 convolutional layer and a pooling layer. After passing through three sets of dense feature extraction modules and corresponding transition modules, the spatial size of the feature maps is reduced to 1 / 16 of the original image.

[0053] Among them, ordinary convolutional layers are used to extract stable local structural features; deformable convolutional layers introduce learnable spatial offsets in the convolution operation, so that the sampling position of the convolution kernel can be adaptively adjusted according to the input features, so as to more flexibly model the irregular shape of the strokes of symbols, the spatial distribution changes between symbols, and the deformation of local structures in handwritten mathematical formulas.

[0054] After completing the dense feature extraction, the size is obtained as follows: The feature map, where .

[0055] Then, a 1×1 convolutional layer is used to map the number of channels to the hidden dimension d_model required by the Transformer decoder, and the feature map is rearranged in dimensions to obtain a size of The feature tensor.

[0056] To enhance the model's ability to perceive spatial location information, a two-dimensional positional encoding is introduced onto the feature tensor to represent the positional information of the features in two-dimensional space, followed by normalization. Finally, the visual encoder outputs a visual feature tensor and its corresponding mask tensor. The visual feature tensor serves as the key and value input to the cross-attention module in the decoder, guiding the subsequent generation of symbol sequences.

[0057] Step 2.2, Decoder construction.

[0058] The decoder is a sequence generation model based on the Transformer architecture, used to map the two-dimensional visual feature sequence output by the visual encoder into a one-dimensional LaTeX symbol sequence. This decoder uses an autoregressive approach to progressively generate the target sequence, enabling it to simultaneously model long-range dependencies between symbols and the correspondence between symbols and image regions.

[0059] Step 2.2.1, Overall architecture of the decoder.

[0060] The decoder is built on the Transformer architecture and consists of multiple identical decoding layers stacked together. In this embodiment, the decoder contains three decoding layers. Each decoding layer contains three sub-modules: a multi-head self-attention module integrating rotational position encoding, a cross-attention module integrating a coverage attention refinement mechanism, and a feedforward neural network.

[0061] Step 2.2.2: Target sequence embedding and mask construction.

[0062] Each symbol in the target LaTeX symbol sequence is first mapped to a d-dimensional vector representation through an embedding layer, resulting in an embedding sequence. Subsequently, character-level positional encodings are added to the embedding vectors to incorporate the symbol's positional information within the sequence. To avoid leaking future information when predicting symbols at the current position, a causal attention mask is constructed. This allows the model to focus only on historical symbols. Simultaneously, a padding mask is constructed for the padding symbols. This avoids invalid locations filled in during the preprocessing stage from participating in attention calculations.

[0063] Step 2.2.3: Integrate a multi-head self-attention module with rotational position encoding.

[0064] In the multi-head self-attention module, the query vector is first obtained by linearly mapping the input representation. Before calculating the attention score, a rotation position encoding transformation is applied to the query and key vectors.

[0065] (1) Definition of angular frequency sequence

[0066] For the feature dimension d, define the angular frequency sequence: .

[0067] (2) Construction of sine and cosine tensors

[0068] For position m, construct the cosine tensor. Sine tensor ,

[0069]

[0070]

[0071] (3) Definition of rotation operation

[0072] Divide the vector into two-dimensional sub-vectors along two adjacent dimensions, and perform an orthogonal rotation on each sub-vector:

[0073]

[0074] Record this operation as .

[0075] (4) Rotational position encoding transformation

[0076] The original vector and the rotated vector are multiplied element-wise by the cosine tensor and the sine tensor, respectively, and then summed to obtain their rotated representations. Here, the rotated representations of the query vector and the key vector are:

[0077]

[0078]

[0079] Where ⊙ represents element-wise multiplication.

[0080] (5) Attention Calculation

[0081] Calculate the attention score using the rotated query and key vectors:

[0082]

[0083] This inner product can be expressed as

[0084]

[0085] in It is a two-dimensional rotation matrix. It is an angular frequency sequence, with the corresponding frequency applied to each dimension pair. .

[0086] Due to the orthogonality of the rotation matrix, the inner product result of this attention depends only on the relative position (nm), thus enabling explicit modeling of the relative positional relationships between arbitrary distance symbols without introducing additional learnable parameters, thereby enhancing the model's ability to understand long formulas and complex two-dimensional spatial structures.

[0087] In a multi-head attention structure, each attention head performs the above calculations in parallel, and finally the outputs of each head are spliced ​​together and fused through a linear layer.

[0088] Step 2.2.4: Integrate the cross-attention module that covers the attention refinement mechanism.

[0089] The cross-attention module aligns the decoded sequence with the output features of the visual encoder. The query vector is generated by the decoder's hidden state at the current time step t, while the key and value vectors are provided by the visual feature sequence output by the visual encoder. Based on the query vector, key vector, and value vector, the original cross-attention score e and its corresponding cross-attention weights at the current time step t are calculated. .

[0090] To prevent repeated attention to already parsed image regions during decoding and to enhance attention to unparsed regions, a coverage attention refinement mechanism is introduced. Let's assume that before time step t, each historical time step... The corresponding cross-attention weights are To address the attention contributions at different historical time steps, a time-weighted coefficient ω(t,i) related to the current time step t is introduced. This time-weighted coefficient is determined based on the time interval (t−i) between the current time step and historical time steps, and is used to differentiate and weight the cross-attention weights of different historical time steps.

[0091] In a preferred embodiment, the time weighting coefficient ω(t,i) is determined by an exponential decay with respect to the time interval, and is defined as follows:

[0092]

[0093] in, The preset decay coefficient is used to control the rate at which the attention weight of the historical time step gradually decreases as the time interval increases.

[0094] Based on this, the historical cross-attention weights, after being weighted by time, are accumulated position by position to obtain the coverage vector corresponding to the current time step. Its calculation method is expressed as follows:

[0095]

[0096] The covering vector is processed through convolutional layers, activation functions, linear mapping, and pooling operations to obtain the covering penalty term. The original attention score e is corrected to:

[0097]

[0098] Again Perform Softmax normalization to obtain the final attention weights. Since the cross-attention weights are obtained by Softmax normalization of the attention scores, adjusting the attention scores is equivalent to adjusting the cross-attention weights.

[0099] Step 2.2.5, Feedforward Neural Network and Residual Structure.

[0100] The feedforward neural network independently performs a nonlinear mapping on the features at each time step, in the form of:

[0101]

[0102] in This is the activation function. This module is used to improve the model's feature representation ability. Residual connections and layer normalization operations are set after self-attention, cross-attention, and feedforward networks.

[0103] Step 2.2.6, Output layer and autoregressive generation.

[0104] The output of the last layer of the decoder is mapped to the target character set dimension via linear projection, and then the conditional probability distribution is obtained through the Softmax function:

[0105]

[0106] The model generates LaTeX symbol sequences stepwise in an autoregressive manner until the end symbol is generated. This process enables the structured recognition of handwritten mathematical formulas.

[0107] Step 3: Construct the loss function and train the model.

[0108] The cross-entropy loss function is constructed as the main training and optimization objective of the handwritten mathematical formula recognition model integrated with a rotation-position encoder-decoder. For each handwritten formula image and its corresponding LaTeX sequence label in the training samples, the decoder outputs the probability distribution of the target character set at each time step. The cross-entropy loss is defined as follows:

[0109]

[0110] in, Let t be the t-th real symbol. Predict probabilities for the model.

[0111] The preprocessed image, mask, and corresponding LaTeX sequence are used together to form the training data, which are input into the model in an end-to-end manner. The backpropagation algorithm is used to jointly optimize the parameters of the visual encoder and the decoder, which includes rotation position encoding. During training, a stochastic gradient descent optimizer is used, and the learning rate is dynamically adjusted based on the performance on the validation set, allowing the model to gradually learn the mapping relationship from image features to symbol sequences.

[0112] Through the above training process, the decoder with integrated rotational position encoding is able to learn the generation rules of symbol sequences and more accurately model the relative positional relationships between symbols.

[0113] Step 4: Reasoning stage - Identify handwritten mathematical formulas.

[0114] During the inference phase, images of handwritten mathematical formulas are input into the trained recognition model to generate LaTeX symbol sequences. In the decoding process, a beam search strategy is employed, retaining candidate sequences with higher probabilities at each step until the end symbol is generated or the maximum length is reached. Finally, the sequence with the highest overall probability is selected as the recognition result output.

[0115] This invention employs a technical solution combining a deep convolutional neural network and a Transformer decoder with integrated rotational position coding. It extracts multi-level features from handwritten mathematical formula images through a visual encoder and introduces rotational position coding during the decoding stage. This allows attention computation to explicitly model the relative positional relationships between symbols, effectively solving the problem of insufficient modeling ability for long formulas and complex nested structures in existing technologies, and improving the model's understanding of the overall formula structure. Furthermore, this invention introduces a coverage attention refinement mechanism to constrain already focused areas, reducing redundant attention and symbol omissions, thus improving the accuracy and stability of the recognition results. This technical solution improves recognition performance without significantly increasing model parameters, exhibits good generalization ability and practical value, and is suitable for applications such as the digitization of mathematical content and intelligent education.

Claims

1. A method for recognizing handwritten mathematical formulas based on rotational position encoding, comprising the following steps: Step 1: Obtain the image of the handwritten mathematical formula and preprocess the image of the handwritten mathematical formula; Step 2: Construct a handwritten mathematical formula recognition model, which includes a visual encoder and a Transformer decoder integrating rotational position encoding; the rotational position encoding is used to perform position-related rotational transformations on the query vector and key vector in the self-attention calculation of the decoder. Step 3: Construct a loss function and train the handwritten mathematical formula recognition model; Step 4: Input the image of the handwritten mathematical formula to be recognized into the trained handwritten mathematical formula recognition model, and output the corresponding LaTeX symbol sequence.

2. The handwritten mathematical formula recognition method based on rotational position encoding according to claim 1, characterized in that, In step 1, the preprocessing of the handwritten mathematical formula image includes: random scale transformation, size normalization, and padding and masking to generate a standardized image tensor that meets the input requirements of the handwritten mathematical formula recognition model based on rotation position encoding.

3. The handwritten mathematical formula recognition method based on rotational position encoding according to claim 1, characterized in that, The handwritten mathematical formula recognition model constructed in step 2 adopts an end-to-end encoder-decoder structure, wherein: the visual encoder extracts two-dimensional visual features from the input handwritten mathematical formula image; the Transformer decoder, guided by the visual features, gradually generates a one-dimensional LaTeX symbol sequence corresponding to the handwritten mathematical formula structure in an autoregressive manner.

4. The handwritten mathematical formula recognition method based on rotational position encoding according to claim 3, characterized in that, The visual encoder uses a deep convolutional neural network as its backbone. It first performs convolution and pooling operations sequentially on the preprocessed handwritten mathematical formula image to extract preliminary features, which are then input into multiple dense feature extraction modules and transition modules. Each dense feature extraction module consists of multiple convolutional layers, including ordinary and deformable convolutional layers, connected via feature concatenation. The transition module performs channel compression and spatial downsampling. After passing through the three dense feature extraction modules and corresponding transition modules, a two-dimensional spatial feature map F is output. This feature map F is flattened in the spatial dimension and mapped to a visual feature sequence, with two-dimensional position encoding introduced to represent the positional information of the features in two-dimensional space. Finally, the visual feature sequence serves as the key and value input to the cross-attention module in the decoder.

5. The handwritten mathematical formula recognition method based on rotational position encoding according to claim 3, characterized in that, The Transformer decoder consists of multiple stacked decoding layers. Each decoding layer includes a multi-head self-attention module integrating rotational position encoding, a cross-attention module integrating a coverage attention refinement mechanism, and a feedforward neural network. In the multi-head self-attention module, rotational position encoding transformations related to the symbol sequence position are first applied to the query vector Q and the key vector K, while keeping the corresponding value vectors unchanged. Dot product attention is then calculated using the query vector and key vector after the rotational position encoding transformation to obtain the self-attention output, which serves as the query input for the cross-attention module in the subsequent decoding layer. The cross-attention module integrating the coverage attention refinement mechanism introduces the accumulated information of the cross-attention weights from historical time steps during the decoding process and corrects the cross-attention score of the current time step based on the accumulated information.

6. The handwritten mathematical formula recognition method based on rotational position encoding according to claim 5, characterized in that, The rotation position encoding is implemented as follows: An angular frequency sequence is defined based on the feature dimension d, and a cosine vector and a sine vector corresponding to the angular frequency sequence are generated based on the position of the symbol in the sequence; the feature dimensions of the query vector and the key vector are divided into d / 2 two-dimensional sub-vectors according to two adjacent elements; for each of the two-dimensional sub-vectors, a rotation transformation is performed according to a preset orthogonal rotation rule to obtain the corresponding rotation vector; the original query vector and the key vector are multiplied element-wise with the cosine vector at the corresponding position, and the rotation vector is multiplied element-wise with the sine vector at the corresponding position, and then the above two results are added element-wise to obtain the query vector and the key vector after rotation position encoding transformation.

7. The handwritten mathematical formula recognition method based on rotational position encoding according to claim 5, characterized in that, The coverage attention refinement mechanism first calculates the cross-attention score and corresponding cross-attention weight at the current time step during the decoding process, and introduces a time-related weighting coefficient for the cross-attention weights of historical time steps. The weighting coefficient is determined based on the time interval between the current time step and the historical time steps. Then, the cross-attention weights of the historical time steps are weighted and accumulated to obtain the coverage vector. Finally, the cross-attention weights of the current time step are corrected based on the coverage vector to refine the cross-attention distribution.

Images