A method for recognizing handwritten mathematical formulas

Abstract

The present invention proposes a handwritten mathematical formula recognition method, which relates to the technical field of optical character recognition. The method comprises constructing collected handwritten formula image samples into an image dataset, then constructing a handwritten mathematical formula recognition model including an encoding module, a counting module, and a decoding module and training the model using the image dataset. The method improves the focusing and decoding capabilities of the model through a bidirectional decoding mechanism and an attention refinement module, and introduces a counting module to further improve the recognition accuracy of handwritten mathematical formulas in a multi-task collaborative manner. The handwritten formula image to be recognized is input into the trained handwritten mathematical formula recognition model, and a LaTeX sequence of the recognized handwritten formula is output, thereby improving the recognition accuracy of handwritten mathematical formulas.

Description

Technical Field

[0001] The present invention relates to the technical field of optical character recognition, and more particularly to a method for recognizing handwritten mathematical formulas. Background Art

[0002] The internet has facilitated the penetration and development of the digital economy across various industries. The rich user behavior generated by the internet has generated vast amounts of data, fueling the rapid development of artificial intelligence. Computer vision and natural language processing are highly sought-after areas within artificial intelligence. Object detection in computer vision, which automatically identifies the location and category of objects, is widely used in a variety of fields, including traffic capture, industrial parts fault detection, and facial recognition. It has significant practical significance for facilitating people's lives and reducing unnecessary human effort.

[0003] The Mathematical Expression Recognition (MER) task involves converting electronic documents or handwritten mathematical formula images into standardized LaTeX sequences using computer vision techniques, through symbol segmentation, recognition, and reorganization algorithms or deep learning methods. The Handwritten Mathematical Expression Recognition (HMER) task, as a branch of the MER task, includes offline handwritten mathematical formula recognition and online handwritten mathematical formula recognition tasks. The offline handwritten mathematical formula recognition task is to recognize images of handwritten mathematical formulas as LaTeX sequences or other standardized formats, using only the image information generated after the user's handwritten formula is written for segmentation, recognition, and other operations. The offline handwritten mathematical formula task is not limited to online handwriting, but can also recognize, aggregate, and organize offline handwritten images (such as blackboard images), helping to achieve the integration of offline resources on the Internet.

[0004] The prior art discloses a method for offline handwritten mathematical formula recognition based on deep learning. The method utilizes the encoder-decoder processing flow of deep learning, uses a multi-scale DenseNet combined with GCA to encode image content, and uses a Transformer to decode the encoded features. At the same time, a bidirectional training strategy and a mutual learning mechanism are introduced to handle the offline handwritten mathematical formula recognition problem end-to-end. A graphical interface is designed to simulate handwriting input with a mouse, and the trained encoder-decoder model is used to recognize handwritten formulas and provide recognized Latex strings. However, in this solution, on the one hand, accurate recognition and counting of symbols cannot be achieved, and on the other hand, character recognition errors can easily be caused by distraction or focus errors. Summary of the Invention

[0005] In order to solve the problem of low character recognition accuracy in current handwritten mathematical formula recognition methods, the present invention proposes a handwritten mathematical formula recognition method, constructs and trains a handwritten mathematical formula recognition model including an encoding module, a counting module and a decoding module, improves the model's focusing and decoding capabilities through a bidirectional decoding mechanism and an attention refinement module, and introduces a counting module to further improve the accuracy of handwritten mathematical formula recognition in a multi-task collaborative manner.

[0006] In order to achieve the above technical effects, the technical solutions of the present invention are as follows:

[0007] A handwritten mathematical formula recognition method, comprising:

[0008] S1. Collect handwritten formula image samples to form an image dataset, and divide the image dataset into a training set, a validation set, and a test set;

[0009] S2. Constructing a handwritten mathematical formula recognition model, the handwritten mathematical formula recognition model includes an encoding module, a counting module, and a decoding module; training the handwritten mathematical formula recognition model using a training set, then evaluating the handwritten mathematical formula recognition model using a validation set, and testing the effectiveness of the handwritten mathematical formula recognition model using a test set to obtain a trained handwritten mathematical formula recognition model;

[0010] S3. Input the handwritten formula image to be recognized into the trained handwritten mathematical formula recognition model, and output the LaTeX sequence of the recognized handwritten formula.

[0011] This technical solution introduces a counting module based on the conventional handwritten mathematical formula recognition model, and uses a multi-task collaborative approach to further improve the accuracy of handwritten mathematical formula recognition.

[0012] Preferably, the data format of the handwritten mathematical formula recognition model is tensor.

[0013] Preferably, in step S1, the step further includes grayscale processing of the handwritten formula image sample.

[0014] Preferably, the encoding module includes a DenseNet encoding module, a sine-cosine position encoding module and an encoding enhancement module connected in sequence;

[0015] The DenseNet encoding module is used to perform preliminary feature extraction on the input handwritten formula image to obtain a preliminary feature map;

[0016] The sine-cosine position encoding is used to embed the position information of the image features in the preliminary feature map, and the position information is obtained by performing 2D convolution on the preliminary feature map;

[0017] Here, the periodicity of sine and cosine is used to ensure that sequences of different lengths can be represented in the space of [-1,1]. Through the design of combining sine and cosine, the relative position distance characteristics between two positions are represented.

[0018] The coding enhancement module is used to enhance the features of the preliminary feature map embedded with the position information.

[0019] Preferably, the encoding enhancement module includes a Transformer encoding layer and a conditional position encoding generator, and the Transformer encoding layer includes a multi-head attention module and a feedforward network; here, the multi-head attention module is used to obtain the importance of feature maps under the global perspective, and the feedforward network is used to add nonlinear information;

[0020] Input the preliminary feature map X output by DenseNet into the Transformer encoding layer, perform linear transformation on the preliminary feature map X, and obtain the query matrix Q, key matrix K, and value matrix V of the preliminary feature map X;

[0021] Using the multi-head attention module, Q, K, and V are linearly transformed h times and then input into the scaled dot product attention. The h times of scaled dot product attention results are spliced ​​together and then linearly transformed once more to obtain the output result X′ of the multi-head attention. The expression formula of X′ is as follows:

[0022] X'=MultiHead(Q,K,V)=COncat(head1,...,head h )W O

[0023] Here, O represents a 1×1 convolution. Here, we use the multi-head attention module to train the weights of Q, K, and V from different angles and obtain multi-angle attention, so as to represent the overall features in multiple dimensions.

[0024] Use the fully connected feedforward network to further aggregate the features of X′ and output the flat sequence x;

[0025] The conditional position encoding generator is used to embed the flattened sequence x output by the Transformer encoding layer into a conditional position encoding.

[0026] Preferably, in the conditional position coding generator, the flat sequence output by the Transformer coding layer is first Reshape into a two-dimensional space Then it is mapped to the two-dimensional image space and reshaped again. The formula is as follows:

[0027]

[0028] Among them, CPE represents the conditional position encoding generator, Represents the reshaping operation, and F represents the mapping operation; here, the conditional position encoding generator is used to realize the joint effect of absolute position encoding and relative position encoding.

[0029] Here, adding relative position information helps to better utilize the positional relationship between each element.

[0030] Preferably, the counting module includes a 3×3 convolutional network branch and a 5×5 convolutional network branch; the 3×3 convolutional network branch includes a 3×3 convolution, a convolutional attention module for reshaping the weights of the channel dimension and the spatial dimension, a 1×1 convolution, a sigmoid function, and a channel summation function connected in sequence; the 5×5 convolutional network branch includes a 5×5 convolution, a convolutional attention module, a 1×1 convolution, a sigmoid function, and a channel summation function connected in sequence;

[0031] The calculation formula of the single branch count vector C of each branch is as follows:

[0032] C = Sum(sigmoid(O(M)))

[0033] Among them, Sum represents a global summation operation in units of channels, O represents a 1×1 convolution, and M represents the input of the branch; the single-branch count vector C is averaged and the count vector of the preliminary feature map is obtained as the counting module prediction result V.

[0034] Here, handwritten mathematical formula recognition is achieved through multi-task collaboration.

[0035] Preferably, in step S2, when training the decoding module, first obtain the LaTeX sequence of the actual formula of the handwritten formula image sample corresponding to the flat sequence x in the image dataset, record it as the real sequence, and divide the real sequence into the longest sequence length L m Perform zero padding to obtain a regular tensor, and then use the embedding layer to reduce the dimension and vectorize the regular tensor.

[0036] Using sine and cosine position encoding to embed position information into input data;

[0037] Adjust the number of channels of the counting module prediction results and copy them L m Adjust the dimension of the counting module prediction result to be consistent with the output dimension of the embedding layer;

[0038] The regular tensor obtained by converting the flat sequence x and its corresponding real sequence, the position information of the sine and cosine position encoding and the adjusted counting module prediction result are added after dimension unification as the overall input in the decoding module training process.

[0039] Here, the carried position information and the number of characters in the sequence are utilized to effectively expand the amount of information and effectively improve the prediction effect of the overall sequence.

[0040] Preferably, the decoding module is composed of a masked self-attention module, a multi-head interactive attention module feedforward layer and a normalization layer alternately connected in series;

[0041] The multi-head interactive attention module includes an interactive attention module, an attention refinement module and a normalization layer connected in sequence; the input of the interactive attention module includes the matrix Q′ output by the masked self-attention module and the matrix K and matrix V obtained by the Transformer encoding layer in the encoding module.

[0042] Preferably, in step S2, the handwritten mathematical formula recognition model is trained until the loss function Loss of the handwritten mathematical formula recognition model converges; the calculation formula of the loss function Loss is as follows:

[0043]

[0044]

[0045] in, is the cross entropy function between the predicted sequence and probability output by the decoding module during training and the true sequence, It represents the L1 smooth loss between the counting module prediction result V and the actual counting statistics v, and λ is a hyperparameter. Here, the L1 smooth loss is used to improve the zero-point non-smoothness problem of the L1 loss, and compared with the L2 loss, the L1 smooth loss is not sensitive to outliers.

[0046] Preferably, when training the decoding module, the permutation invariance of the decoding module is used to perform bidirectional decoding of the sequence, thereby increasing the effective information in the reverse direction;

[0047] The cross entropy function obtained by the decoding module through bidirectional decoding The calculation formula is as follows:

[0048]

[0049] in, Indicates sequence decoding from left to right, Indicates right-to-left sequence decoding.

[0050] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0051] The present invention proposes a handwritten mathematical formula recognition method. The collected handwritten formula image samples are constructed into an image dataset. Then, a handwritten mathematical formula recognition model including an encoding module, a counting module, and a decoding module is constructed and trained using the image dataset. The focusing and decoding capabilities of the model are improved through a bidirectional decoding mechanism and an attention refinement module. The counting module is introduced to further improve the handwritten mathematical formula recognition accuracy in a multi-task collaborative manner. The handwritten formula image to be recognized is input into the trained handwritten mathematical formula recognition model, and a LaTeX sequence of the recognized handwritten formula is output, thereby improving the recognition accuracy of handwritten mathematical formulas. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic diagram showing a flow chart of a handwritten mathematical formula recognition method proposed in Example 1 of the present invention;

[0053] Figure 2 The figure shows the overall structure of the handwritten mathematical formula recognition model proposed in Example 1 of the present invention;

[0054] Figure 3 Schematic diagram showing the structure of the coding enhancement module proposed in Example 1 of the present invention;

[0055] Figure 4 A schematic diagram showing the structure of a counting module proposed in Example 2 of the present invention;

[0056] Figure 5 Schematic diagram of the structure of the attention refinement module proposed in Example 3 of the present invention. DETAILED DESCRIPTION

[0057] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent;

[0058] In order to better illustrate this embodiment, some parts of the drawings may be omitted, enlarged, or reduced, and do not represent the actual size;

[0059] It is understandable to those skilled in the art that descriptions of certain well-known contents may be omitted in the drawings.

[0060] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0061] The positional relationships described in the drawings are for illustrative purposes only and should not be construed as limiting this patent;

[0062] Example 1

[0063] like Figure 1 As shown, this embodiment proposes a handwritten mathematical formula recognition method, comprising the following steps:

[0064] S1. Collect handwritten formula image samples to form an image dataset, and divide the image dataset into a training set, a validation set, and a test set;

[0065] In this embodiment, the handwritten formula image sample is gray-scale processed to obtain an H×W×1 handwritten formula image.

[0066] S2. Build a handwritten mathematical formula recognition model, such as Figure 2 As shown, the handwritten mathematical formula recognition model includes an encoding module, a counting module and a decoding module; the handwritten mathematical formula recognition model is trained using a training set, and then the handwritten mathematical formula recognition model is evaluated using a validation set, and the effectiveness of the handwritten mathematical formula recognition model is tested using a test set to obtain a trained handwritten mathematical formula recognition model;

[0067] S3. Input the handwritten formula image to be recognized into the trained handwritten mathematical formula recognition model, and output the LaTeX sequence of the recognized handwritten formula.

[0068] The encoding module includes a DenseNet encoding module, a sine-cosine position encoding module and an encoding enhancement module connected in sequence;

[0069] The DenseNet encoding module is used to perform preliminary feature extraction on the input handwritten formula image to obtain a preliminary feature map. In this embodiment, for the input H×W×1 handwritten formula image, it is first initialized through a convolutional network with a convolution kernel of 7×7 and a step size of 2, and then undergoes batch normalization, ReLU function activation, and maximum pooling processing in sequence to obtain preliminary nonlinear information and local information of the image, and then enters the dense connection layer and transition layer of DenseNet. In actual application, the dense connection layer is the bottleneck layer, that is, the BN-ReLU-Conv (1×1)-BN-ReLU-Conv (3×3) structure DenseNet-B, which is beneficial to reducing the amount of calculation when the network deepens and the number of layers increases; after the 3×3 convolution, the output feature map is spliced ​​with the feature map before the dense connection layer in the channel dimension, retaining the features under different receptive fields. In this embodiment, the DenseNet network superimposes three dense connection layers, sets the growth rate of the dense connection layer to 24, performs batch normalization on the feature map, and finally performs linear mapping to output a feature map with 256 channels.

[0070] The sine-cosine position coding is used to embed the position information of the image features in the preliminary feature map, and the position information is obtained by performing 2D convolution on the preliminary feature map, wherein the convolution kernel size of the 2D convolution is greater than 3.

[0071] In this embodiment, sine and cosine coding is used to embed the position information of the image features. The formula is as follows:

[0072]

[0073]

[0074] Among them, PE(pox,2i) and PE(pos,2i+1) represent the 2i and 2i+1 components of the pos position respectively, d model Represents the dimension of the vector in the model. Here, the periodicity of sine and cosine functions is utilized to ensure that sequences of different lengths can be represented in the space [-1, 1]. In addition, considering that the positional encoding of two positions l and l+k separated by k words is a linear transformation defined by the positional encodings of l and k, the relative position distance feature between the two positions is represented by a design in which half of the channel positions are represented by sine functions and half by cosine functions.

[0075] The coding enhancement module is used to enhance the features of the preliminary feature map embedded with the position information, wherein, Figure 3 As shown, the encoding enhancement module includes a Transformer encoding layer and a conditional position encoding generator;

[0076] In this embodiment, the Transformer encoding layer includes a multi-head attention module and a feedforward network. The multi-head attention module is used to obtain the importance of feature maps under the global perspective, and the feedforward network is used to add nonlinear information.

[0077] In this embodiment, the preliminary feature map X output by DenseNet is input into the Transformer encoding layer, and a linear transformation is performed on the preliminary feature map X to obtain the query matrix Q, key matrix K, and value matrix V of the preliminary feature map X. The formula is as follows:

[0078] Q=XW Q

[0079] K=XW K

[0080] V=XW V

[0081] Calculate the similarity by multiplying the query Q with all the keys K, and then divide by The weight of the value V obtained by applying the softmax function is the attention weight value, which is used to obtain the measure of attention of the important area of ​​the feature map Attention (Q, K, V), which is used as the head value in the subsequent multi-head attention formula:

[0082]

[0083] Using the multi-head attention module, Q, K, and V are linearly transformed h times and then input into the scaled dot product attention. In this embodiment, the multi-head attention is set to head = 8; the h times scaled dot product attention results are spliced ​​and then linearly transformed to obtain the output result X′ of the multi-head attention. The expression formula of X′ is as follows:

[0084] X′=MultiHead(Q,K,V)=Concat(head1,...,head h )W O

[0085] Here, O represents a 1×1 convolution. Here, we use the multi-head attention module to train the weights of Q, K, and V from different angles and obtain multi-angle attention, so as to represent the overall features in multiple dimensions.

[0086] A fully connected feedforward network is used to further aggregate features of X′ and output a flattened sequence x. In this embodiment, the feedforward network includes two linear transformations and an activation function:

[0087] FFN(X′)=max(0,X′W1+b1)W2+b2

[0088] In this embodiment, a conditional position encoding generator is used to embed the flat sequence x output by the Transformer encoding layer into a conditional position encoding.

[0089] In the conditional positional encoding generator, the flattened sequence output by the Transformer encoding layer is first Reshape into a two-dimensional space Then it is mapped to the two-dimensional image space and reshaped again. The formula is as follows:

[0090]

[0091] Among them, CPE represents the conditional position encoding generator, Represents the reshaping operation of the tensor, and F represents the mapping operation; here, the conditional position encoding generator is used to realize the joint effect of absolute position encoding and relative position encoding.

[0092] In actual application, the absolute position information mainly comes from the original feature Figure 4 The feature map after 0 padding will have accurate edge recognition after convolution; the generating function F can be implemented by an ordinary two-dimensional convolutional network, a depth-separable convolutional network or other complex structures. In this embodiment, a convolutional network with a convolution kernel of 3×3 is used as the generating function F of the conditional position encoding generator.

[0093] Example 2

[0094] In this embodiment, if Figure 4 As shown, the counting module includes a 3×3 convolutional network branch and a 5×5 convolutional network branch; the 3×3 convolutional network branch includes a 3×3 convolution, a convolutional attention module for reshaping the weights of the channel dimension and the spatial dimension, a 1×1 convolution, a sigmoid function, and a channel summation function connected in sequence; the 5×5 convolutional network branch includes a 5×5 convolution, a convolutional attention module, a 1×1 convolution, a sigmoid function, and a channel summation function connected in sequence;

[0095] The calculation formula of the single branch count vector C of each branch is as follows:

[0096] C = Sum(sigmoid(O(M)))

[0097] Among them, Sum represents a global summation operation in units of channels, O represents a 1×1 convolution, and M represents the input of the branch; the single-branch count vector C is averaged and the count vector of the preliminary feature map is obtained as the counting module prediction result V.

[0098] Here, handwritten mathematical formula recognition is achieved through multi-task collaboration.

[0099] Example 3

[0100] In this embodiment, when training the decoding module, first obtain the LaTeX sequence of the actual formula of the handwritten formula image sample corresponding to the flat sequence x in the image dataset, record it as the real sequence, and then divide the real sequence into the longest sequence length L m Perform zero padding to obtain a regular tensor, and then use the embedding layer to reduce the dimension and vectorize the regular tensor.

[0101] The position information is embedded into the input data using sine-cosine position coding. In the specific operation process, the embedding method is to add the input data and the encoded position information.

[0102] Input the prediction results of the counting module into the fully connected layer, adjust the number of channels to 256 layers, and copy them to L m parts, so that the dimension is unified to be consistent with the output dimension of the embedding layer;

[0103] The warped tensor obtained by converting the flattened sequence x and its corresponding real sequence, the position information of the sine and cosine position encodings, and the adjusted counting module prediction results are added together after dimension unification and serve as the overall input for the decoding module training process. After training, the trained handwritten mathematical formula recognition model no longer requires the warped tensor of the real sequence when it is put into use.

[0104] Here, the carried position information and the number of characters in the sequence are utilized to effectively expand the amount of information and effectively improve the prediction effect of the overall sequence.

[0105] In this embodiment, the decoding module is composed of a masked self-attention module, a multi-head interactive attention module feedforward layer and a normalization layer alternately connected in series;

[0106] The multi-head interactive attention module includes an interactive attention module, an attention refinement module and a normalization layer connected in sequence; the input of the interactive attention module includes the matrix Q output by the masked self-attention module ′ And the matrix K and matrix V obtained by the Transformer encoding layer in the encoding module. In this embodiment, the interactive attention module uses the attention refinement mechanism to allocate and focus the coverage area of ​​attention.

[0107] In this embodiment, see Figure 5 The attention refinement module uses the previously calculated attention weight value to detect the edge of the parsed area, that is, the refined matrix R; the attention matrix E is used to subtract the parsed area to obtain the refined unresolved area attention weight as the area that needs to be focused on in the subsequent time step; the structure of the attention refinement module is composed of the function Indicates that, including matrix reshaping, convolution kernel size is k c The convolution, nonlinearization and normalization process of the convolution network reduces the number of channels and reasonably reduces the overhead brought by the module; the solution process of the refined matrix R can be expressed as:

[0108]

[0109]

[0110]

[0111] Where C represents the cumsum operation on A, that is, the cumulative sum, reshape is the reshaping operation that converts the dimension of T×L×h into T×h0×w0×h, and K represents the convolution kernel size k c The convolution operation, b c represents the bias term, W c Represents a linear mapping matrix.

[0112] The unresolved region after the attention refinement module is represented as:

[0113]

[0114] in:

[0115]

[0116] An attention refinement mechanism combining self-coverage and cross-coverage is applied to model the coverage information to optimize the attention part of the Transformer decoder.

[0117] For self-covering:

[0118]

[0119] For cross coverage:

[0120]

[0121] Combining the two covers together gives us the fused cover:

[0122]

[0123] in Represents the concatenation of two tensors.

[0124] In this embodiment, the decoder outputs the optimal solution of the sequence through beam search, that is, when making predictions at each time step, f optimal solutions are retained as candidate sequences, and finally the sequence with the largest joint probability is selected from the f solutions as the final recognition result.

[0125] During the training process, a loss function Loss of the handwritten mathematical formula recognition model is constructed, and the handwritten mathematical formula recognition model is trained until the loss function Loss of the handwritten mathematical formula recognition model converges; the calculation formula of the loss function Loss is as follows:

[0126]

[0127]

[0128] in, is the cross entropy function between the predicted sequence and probability output by the decoding module during training and the true sequence, It represents the L1 smooth loss between the counting module prediction result V and the actual counting statistics v, and λ is a hyperparameter. Here, the L1 smooth loss is used to improve the zero-point non-smoothness problem of the L1 loss, and compared with the L2 loss, the L1 smooth loss is not sensitive to outliers.

[0129] Preferably, when training the decoding module, the permutation invariance of the decoding module is used to perform bidirectional decoding of the sequence, and the permutation invariance of the Transformer decoder is used to directly embed the bidirectional sequence into the input decoder for decoding, so that both L2R and R2L directions are trained and predicted simultaneously without losing simplicity; and effective information in the reverse direction is increased;

[0130] The cross entropy function obtained by the decoding module through bidirectional decoding The calculation formula is as follows:

[0131]

[0132] in, Indicates sequence decoding from left to right, Indicates right-to-left sequence decoding.

[0133] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A method for recognizing handwritten mathematical formulas, characterized in that: include: S1. Collect handwritten formula image samples to form an image dataset, and divide the image dataset into a training set, a validation set, and a test set; S2. Construct a handwritten mathematical formula recognition model, which includes an encoding module, a counting module, and a decoding module; the encoding module includes a DenseNet encoding module, a sine-cosine position encoding module, and an encoding enhancement module connected in sequence; the encoding enhancement module includes a Transformer encoding layer and a conditional position encoding generator; in the conditional position encoding generator, the flat sequence output by the Transformer encoding layer is first converted to Reshape into a two-dimensional space Then it is mapped to the two-dimensional image space and reshaped again. The formula is as follows: Among them, CPE represents the conditional position encoding generator, Represents a reshape operation, and F represents a mapping operation; the counting module includes a 3×3 convolutional network branch and a 5×5 convolutional network branch; the 3×3 convolutional network branch includes a 3×3 convolution, a convolutional attention module, a 1×1 convolution, a sigmoid function, and a channel summation function connected in sequence; the 5×5 convolutional network branch includes a 5×5 convolution, a convolutional attention module, a 1×1 convolution, a sigmoid function, and a channel summation function connected in sequence; The calculation formula of the single branch count vector C of each branch is as follows: C = Sum(sigmoid(O(M))) Among them, Sum represents a global summation operation in units of channels, O represents a 1×1 convolution, and M represents the input of the branch; the single-branch count vector C is averaged and the count vector of the preliminary feature map is obtained as the prediction result V of the counting module; The handwritten mathematical formula recognition model is trained using the training set, then evaluated using the validation set, and the effectiveness of the handwritten mathematical formula recognition model is tested using the test set to obtain a trained handwritten mathematical formula recognition model. S3. Input the handwritten formula image to be recognized into the trained handwritten mathematical formula recognition model, and output the LaTeX sequence of the recognized handwritten formula.

2. The handwritten mathematical formula recognition method according to claim 1, characterized in that: In step S1, the handwritten formula image sample is also grayscale processed.

3. The handwritten mathematical formula recognition method according to claim 2, characterized in that: The DenseNet encoding module is used to perform preliminary feature extraction on the input handwritten formula image to obtain a preliminary feature map; The sine-cosine position encoding is used to embed the position information of the image features in the preliminary feature map, and the position information is obtained by performing 2D convolution on the preliminary feature map; The coding enhancement module is used to enhance the features of the preliminary feature map embedded with the position information.

4. The handwritten mathematical formula recognition method according to claim 3, characterized in that: The Transformer encoding layer includes a multi-head attention module and a feedforward network; After embedding the preliminary feature map X output by DenseNet into the sine-cosine positional encoding, it is input into the Transformer encoding layer, and a linear transformation is performed on the preliminary feature map X to obtain the query matrix Q, key matrix K, and value matrix V of the preliminary feature map X; Using the multi-head attention module, Q, K, and V are linearly transformed h times and then input into the scaled dot product attention. The h times of scaled dot product attention results are spliced ​​together and then linearly transformed once more to obtain the output result X′ of the multi-head attention. The expression formula of X′ is as follows: X′=MultiHead(Q,K,V)=Concat(head1,...,head h )W O Among them, W O represents a linear layer of dimension h×h; Use the fully connected feedforward network to further aggregate the features of X′ and output the flat sequence x; The flattened sequence x is embedded into a conditional positional encoding using the conditional positional encoding generator.

5. The handwritten mathematical formula recognition method according to claim 4, characterized in that: In step S2, when training the decoding module, first obtain the LaTeX sequence of the actual formula of the handwritten formula image sample corresponding to the flat sequence x in the image dataset, record it as the real sequence, and divide the real sequence into the longest sequence length L m Perform zero padding to obtain a regular tensor, and then use the embedding layer to reduce the dimension and vectorize the regular tensor. Using sine and cosine position encoding to embed position information into input data; Adjust the number of channels of the counting module prediction results and copy them L m Adjust the dimension of the counting module prediction result to be consistent with the output dimension of the embedding layer; The regular tensor obtained by converting the flat sequence x and its corresponding real sequence, the position information of the sine and cosine position encoding and the adjusted counting module prediction result are added after dimension unification as the overall input in the decoding module training process.

6. The handwritten mathematical formula recognition method according to claim 5, characterized in that: The decoding module consists of a masked self-attention module, a multi-head interactive attention module feedforward layer and a normalization layer alternately connected in series; The multi-head interactive attention module includes an interactive attention module, an attention refinement module and a normalization layer connected in sequence; the input of the interactive attention module includes the matrix Q output by the masked self-attention module ′ And the matrix K and matrix V obtained by the Transformer encoding layer in the encoding module.

7. The handwritten mathematical formula recognition method according to claim 6, characterized in that: In step S2, the handwritten mathematical formula recognition model is trained until the loss function Loss of the handwritten mathematical formula recognition model converges; the calculation formula of the loss function Loss is as follows: Among them, θ represents the network parameters that need to be updated, is the cross entropy function between the predicted sequence and probability output by the decoding module during training and the true sequence, Represents the L1 smoothing loss between the counting module prediction result V and the actual counting statistics v, and λ is a hyperparameter.

8. The handwritten mathematical formula recognition method according to claim 7, characterized in that: When training the decoding module, the permutation invariance of the decoding module is used to perform bidirectional decoding of the sequence; The cross entropy function obtained by the decoding module through bidirectional decoding The calculation formula is as follows: Among them, L represents the length of the decoding sequence, Z represents the size of a Batch, Indicates sequence decoding from left to right, Indicates right-to-left sequence decoding.

Images