A multi-scale recognition method for handwritten mathematical formulas based on counting module

Abstract

A multi-scale recognition method for handwritten mathematical formulas based on a counting module relates to the field of image recognition technology and solves the problem of low recognition accuracy of handwritten digit recognition models in the prior art. The method constructs a handwritten mathematical formula image dataset, inputs the handwritten mathematical formula image into a handwritten mathematical formula recognition model based on a counting module, uses ResNet to extract feature maps of the handwritten mathematical formula images in the processed image dataset, uses a symbol counting module to add global information to the feature maps, and outputs predicted LaTeX sequences through LSTM. The method models handwritten mathematical formula recognition as an image-to-sequence task. An encoder is used to extract features of the input image, and a decoder gradually predicts the corresponding LaTeX sequence. The method effectively transforms the handwritten mathematical formula recognition problem into an end-to-end sequence generation problem, simplifying the complexity of the problem.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a multi-scale recognition method for handwritten mathematical formulas based on a counting module. Background Art

[0002] In recent years, researchers have developed a variety of mathematical formula recognition models, among which the encoder-decoder structure is widely used. These handwritten formula recognition models usually use convolutional neural networks (CNN) as encoders to map mathematical formula images to high-level feature representations, and decoders usually use recurrent neural networks (RNN) to convert high-level features into output sequences word by word. In the character prediction task, the attention mechanism embedded in the decoder can scan the entire input mathematical formula image and select the most relevant area for decoding.

[0003] Traditional HMER methods usually include three main steps: symbol segmentation, symbol recognition, and structural analysis. In these methods, symbol recognition requires symbol segmentation, which brings many challenges in practical applications. In addition, structural analysis requires prior knowledge of the grammar of mathematical formulas, and the complexity of the parsing algorithm increases exponentially with the complexity of the predefined grammar.

[0004] The encoder based on CNN is the most widely used in the encoder-decoder model. Existing literature 1: Handwritten mathematical expression recognition method based on end-to-end neural network [J]. Pattern Recognition, 2017, 71: 196-206. A model based on the encoder-decoder structure is proposed to solve the offline handwritten mathematical formula recognition task and obtain good results, but most researchers study how to optimize the decoder. For example, existing literature 2: Image to image: accurate and interpretable online handwritten mathematical expression recognition, [C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 35(4): 2925-2933. In this paper, the CNN decoder is used to solve the gradient vanishing and gradient exploding problems. However, these studies ignore the problem of detail feature jump and loss in the DenseNet encoder, especially in the handwritten formula recognition task, this problem becomes more obvious.

[0005] At present, most encoder-decoder models use ordinary CNN or RNN for encoding and decoding operations. However, due to the diversity of handwritten characters, some subtle features will be lost after multiple pooling of convolutional neural networks during the encoding process, such as decimal points, and gradients will disappear or explode. In addition, simple RNN decoders cannot focus on local features for decoding operations, and thus cannot accurately achieve structural analysis and symbol recognition.

[0006] Based on the problem of low recognition accuracy in current recognition models, the present invention proposes a model method that uses a residual network (Residual Network, ResNet) as an encoder to extract features from images, and combines a long short-term memory network (Long Short-Term Memory, LSTM) with a counting module as a decoder. In this way, a single CNN and RNN model is replaced. When decoding LaTeX sequences, the counting module is introduced, and the counting module can guide the model to accurately realize symbol recognition and structural analysis based on the global information of the image. Summary of the invention

[0007] The present invention aims to solve the problem of low recognition accuracy of handwritten digit recognition models in the prior art and provides a multi-scale recognition method for handwritten mathematical formulas based on a counting module.

[0008] A multi-scale recognition method for handwritten mathematical formulas based on a counting module is implemented by the following steps:

[0009] Step 1: Obtain a dataset of handwritten mathematical formula images;

[0010] Step 2: Perform multi-scale scaling and augmentation processing on the handwritten mathematical formula image dataset described in step 1 to obtain a processed image dataset;

[0011] Step 3: Construct a handwritten mathematical formula recognition model based on a counting module, use the ResNet network to extract feature maps from the handwritten mathematical formula images in the processed image data set, use the symbol counting module to add global information to the feature maps, and finally output the predicted LaTeX sequence through the LSTM network; the specific process is:

[0012] Step 31: Use the ResNet network as an encoder to extract features from the input handwritten mathematical formula image to obtain a feature map F;

[0013] Step 32: Use the symbol counting module to extract symbol features from the feature graph F to obtain a symbol counting vector V, wherein each value in the symbol counting vector V is a count value of a predicted corresponding symbol;

[0014] Step 33: Use the LSTM network as a decoder to decode the feature map F obtained in step 31, and at the same time input the symbol count vector V obtained in step 32 into the decoder as global information, and finally generate a LaTeX formula sequence corresponding to the handwritten mathematical formula image.

[0015] The beneficial effects of the present invention are:

[0016] 1. In the method of the present invention, a multi-scale scaling and augmentation method is used to scale and augment the handwritten formula image to increase the scale diversity of the training data. By randomly adjusting the size of the handwritten formula image before training, the decoding and recognition robustness of the handwritten mathematical formula recognition model for symbols of different scales is improved;

[0017] 2. In the method of the present invention, a recognition model based on an encoder-decoder framework is proposed. The model models the recognition of handwritten mathematical formulas as an image-to-sequence task. In this process, the encoder is used to extract the features of the input image, while the decoder gradually predicts the corresponding LaTeX sequence. The handwritten mathematical formula recognition problem is effectively converted into an end-to-end sequence generation problem, thereby simplifying the complexity of the problem;

[0018] 3. In the method of the present invention, in order to solve the problem of inaccurate attention, it is proposed to add a counting module to make the attention more accurate. At the same time, the counting result can represent the number of symbols and can be used as additional global information to improve the accuracy of recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is an effect diagram of a mixed handwritten mathematical formula image data set in a multi-scale recognition method for handwritten mathematical formulas based on a counting module described in the present invention.

[0020] Figure 2 Schematic diagram of multi-scale image scaling.

[0021] Figure 3 This is a flow chart of a multi-scale recognition method for handwritten mathematical formulas based on a counting module described in the present invention.

[0022] Figure 4 Schematic diagram of the handwritten mathematical formula recognition model based on the counting module.

[0023] Figure 5 Schematic diagram of the structure of the ResNet encoder.

[0024] Figure 6 It is a structural diagram of the counting module.

[0025] Figure 7 Schematic diagram of the structure of the LSTM decoder.

[0026] Figure 8 This is a comparison diagram of the models in the existing CROHME2019 dataset and the method of the present invention. DETAILED DESCRIPTION

[0027] Combination Figures 1 to 8 This embodiment describes a multi-scale recognition method for handwritten mathematical formulas based on a counting module, which is implemented by the following steps:

[0028] Step 1: Obtain the CROHME2019 handwritten mathematical formula image data to make a dataset, and divide the image dataset into a training set and a test set at a ratio of 9:1. The training set is used to train the handwritten mathematical formula recognition model based on the counting module constructed in step 3, and the test set is used to test the accuracy of the model. Figure 1 As shown, the dataset contains 1199 handwritten math formula images and corresponding Latex format math formulas, as well as 111 types of symbols, including the start symbol "sta" and the end symbol "end";

[0029] Step 2: Perform multi-scale scaling and augmentation processing on the training set in the handwritten mathematical formula image dataset, by adjusting the two-dimensional coordinate points in the pixel trajectory in the handwritten mathematical formula image The ratio of , in the horizontal and vertical directions to achieve proportional scaling augmentation. The specific implementation is as follows:

[0030] ;

[0031] In the formula, , represents the new coordinate value after scaling and augmentation, k represents the randomly generated scaling factor, satisfying Uniform distribution of range. is a hyperparameter that controls the random scaling ratio and its value is set to 0.2.

[0032] like Figure 2 As shown in the figure, the original image and the image after multi-scale scaling are shown. After scaling, the image will be padded to the set size to ensure that its size is consistent with the size used in the training phase. By applying this scaling augmentation technology, not only the scale variation of the training data is enriched, but also it helps to improve the model's feature extraction ability when identifying symbols of different sizes, and enhances the model's ability to recognize multi-scale objects;

[0033] Step 3: Construct a handwritten mathematical formula recognition model based on a counting module, and use the model to recognize LaTeX sequences from handwritten mathematical formula images; in the model, ResNet is used as an encoder to extract feature maps from mathematical formula images in the input training set, and then a symbol counting module is used to add additional global information to the feature maps, and finally the LSTM recurrent neural network is used as a decoder to gradually output the predicted LaTeX sequence; Figure 4 As shown, the specific implementation process is:

[0034] Step 31: Use the ResNet as an encoder to extract features from the input handwritten mathematical formula image to obtain a feature map F;

[0035] like Figure 5 As shown, in this embodiment, a convolutional neural network with a 33-layer ResNet structure is used as an encoder to extract feature information from the input handwritten mathematical formula image. The encoder is composed of 33 ResNet modules, each of which is composed of a basic convolutional layer with a convolution kernel of 3×3 and a step size of 1, an intermediate batch normalization layer (Batch Norm), a ReLU activation layer, a 5×5 convolutional layer and a bottom batch normalization layer (Batch Norm). The handwritten mathematical formula image is input into the encoder, and after the features are extracted by the first ResNet module, the features are used as the input of the next ResNet module. By analogy, the input image is feature extracted through 33 layers of ResNet modules, and a convolutional layer with a convolution kernel size of 3×3 and a fully connected layer are added after the last ResNet module. Finally, a three-dimensional feature vector of size H×W×D is output, namely: feature map F; where H, W, and D are the height, width, and number of channels of feature map F respectively;

[0036] Step 32: Use the symbol counting module to extract symbol features from the feature graph F to obtain a symbol counting vector V, wherein each value in the symbol counting vector V is a count value of a predicted corresponding symbol;

[0037] like Figure 6 As shown, in this embodiment, the symbol counting module is composed of a 5×5 convolution layer with a convolution kernel, a channel attention module, a 1×1 convolution layer and a Sigmoid activation function; the process of the symbol counting module processing the feature map F is:

[0038] Step A1: A convolution layer with a convolution kernel of 5×5 is used to extract preliminary symbol features from the feature map F. A channel attention module consisting of a global average pooling layer, two linear layers and a sigmoid activation function is used to learn the relationship between channels separately and assign different channel weights to the preliminary symbol features to obtain enhanced symbol features.

[0039] Step A2: perform an element-wise multiplication operation on the preliminary symbol feature and the enhanced symbol feature, and fuse the enhanced symbol feature and the preliminary symbol feature to obtain a symbol feature map;

[0040] Step A3: Send the symbol feature map into a 1×1 convolution to reduce the channel dimension to 111 dimensions. At this time, the number of channels of 111 dimensions represents the types of all symbols in the handwritten mathematical formula image.

[0041] Step A4: Use the Sigmoid activation function to compress each element value in the symbol feature map after dimensionality reduction to the range of (0, 1) to obtain a symbol density feature map;

[0042] Step A5, after adding up all the elements of each dimension, a 111-dimensional symbol count vector V is obtained, and each value in the count vector V is the count value of the predicted corresponding symbol;

[0043] Step 33: Use the LSTM network as a decoder to decode the feature map F obtained in step 31, and at the same time input the symbol count vector V obtained in step 32 into the decoder as global information, and finally generate a LaTeX formula sequence corresponding to the handwritten mathematical formula image.

[0044] like Figure 7 As shown, in this embodiment, the LSTM network as a decoder is composed of a forget gate, an input gate and an output gate; the decoder decodes the feature map F of step three-one, that is, the decoder divides the input feature map F according to the channel dimension to obtain a feature vector sequence, ,in, is the channel dimension of the feature map F at the current time step t;

[0045] The elements in the feature vector sequence X According to the time step input LSTM network to calculate the hidden state at the corresponding time step t and context vector During the decoding process, the symbol count vector V obtained in step 3.2 is input into the decoder as additional global information to assist in generating the LaTeX formula sequence corresponding to the handwritten mathematical formula image; the specific process is as follows:

[0046] Step B1: Forget the context vector of the previous time step t-1 through the forget gate Selectively forget, save important features, and forget low-value features. The final output of the forget gate is , the formula is as follows:

[0047] ,in, represents the Sigmoid activation function, , They are two trainable weights for the forget gate;

[0048] Step B2: The hidden state of the previous time step t-1 is input through the input gate and the input feature vector Perform feature screening and finally input gate output and , the calculation formula is as follows:

[0049] ;

[0050] ;

[0051] In the formula, , , , Four trainable weights for the input gate respectively;

[0052] Step B3: Calculate the hidden state of the output gate result of the LSTM network at the current time step t and context vector ; The calculation formula is as follows:

[0053] ;

[0054] ;

[0055] ;

[0056] In the formula, It means to calculate the Hadamard product of two vectors. is the intermediate variable of the output gate, , Two trainable weights for the output gate respectively;

[0057] Step B4: Get the hidden state of the current time step t and context vector After that, the context vector is transformed into , hidden state Concatenate with the count vector V to predict the predicted sequence symbol at the current time step t and the corresponding symbol prediction probability The specific formula is as follows:

[0058] ;

[0059] ;

[0060] In the formula, Represents a linear transformation function followed by a softmax activation function, which is used to output the symbol prediction probability of the current time step t. Argmax represents the symbol prediction probability Output the corresponding symbol;

[0061] Step B5: When the predicted sequence symbol After predicting the termination symbol "end", concatenate the predicted sequence symbols of all predicted time steps in order to obtain the final output sequence :

[0062] ;

[0063] In the formula, Indicates that the termination symbol "end" is predicted at the current time step t.

[0064] The multi-scale recognition method described in this embodiment is compared with other model methods using the official CROHME2019 dataset, such as Figure 8 As shown in Table 1 below,

[0065] Table 1

[0066]

[0067] The expression recognition accuracy of the method described in this embodiment on the CROHME 2019 dataset, Symbol error indicator and The symbol error indexes are 59.82%, 74.77% and 82.43% respectively, which are significantly ahead of the above methods. This shows that the method proposed in the present invention has a good effect in handwritten mathematical formula recognition.

[0068] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0069] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A multi-scale recognition method for handwritten mathematical formulas based on a counting module, characterized by: The method is implemented by the following steps: Step 1: Obtain a dataset of handwritten mathematical formula images; Step 2: Perform multi-scale scaling and augmentation processing on the handwritten mathematical formula image dataset described in step 1 to obtain a processed image dataset; Step 3: Construct a handwritten mathematical formula recognition model based on a counting module, use the ResNet network to extract feature maps from the handwritten mathematical formula images in the processed image data set, use the symbol counting module to add global information to the feature maps, and finally output the predicted LaTeX sequence through the LSTM network; the specific process is: Step 31: Use the ResNet network as an encoder to extract features from the input handwritten mathematical formula image to obtain a feature map F; Step 32: Use the symbol counting module to extract symbol features from the feature graph F to obtain a symbol counting vector V, wherein each value in the symbol counting vector V is a count value of a predicted corresponding symbol; Step 33: Use the LSTM network as a decoder to decode the feature map F obtained in step 31, and input the symbol count vector V obtained in step 32 into the decoder as global information, and finally generate a LaTeX formula sequence corresponding to the handwritten mathematical formula image; The symbol counting module is composed of a 5×5 convolution layer, a channel attention module, a 1×1 convolution layer and a Sigmoid activation function; the process of the symbol counting module processing the feature map F is as follows: Step A1, using a convolution layer with a convolution kernel of 5×5 to extract preliminary symbol features from the feature map F, and using the channel attention module to learn the preliminary symbol features to obtain enhanced symbol features; Step A2: perform an element-wise multiplication operation on the preliminary symbol feature and the enhanced symbol feature to achieve fusion of the enhanced symbol feature and the preliminary symbol feature, and obtain a symbol feature map; Step A3: sending the symbol feature map to a 1×1 convolutional layer to reduce the channel dimension to 111 dimensions, thereby achieving dimensionality reduction of the symbol feature map; The number of channels in the 111-dimensional image is used as the types of all symbols in the handwritten mathematical formula image; Step A4: Using the Sigmoid activation function, each element value in the symbol feature map after dimensionality reduction is compressed to the range of (0, 1) to obtain a symbol density feature map; Step A5: Add all elements of each dimension in the symbol density feature map to obtain a 111-dimensional symbol count vector V, where each value in the symbol count vector V is the predicted count value of the corresponding symbol.

2. The multi-scale recognition method for handwritten mathematical formulas based on a counting module according to claim 1 is characterized in that: In step 2, multi-scale scaling and augmentation processing is performed on the handwritten mathematical formula image dataset, specifically: the scale of the two-dimensional coordinate points in the pixel trajectory in the handwritten mathematical formula image is adjusted to achieve proportional scaling and augmentation in the horizontal and vertical directions.

3. The multi-scale recognition method for handwritten mathematical formulas based on a counting module according to claim 1 is characterized in that: The encoder is composed of multiple ResNet modules, each of which is composed of a basic convolution layer with a convolution kernel of 3×3 and a step size of 1, an intermediate batch normalization layer, a ReLU activation layer, a 5×5 convolution layer and a bottom batch normalization layer; The handwritten mathematical formula image is input into the encoder. After the features are extracted through the first ResNet module, the features are used as the input of the next ResNet module. The input image is subjected to feature extraction through multiple ResNet modules in sequence. After the last ResNet module, a convolution layer with a convolution kernel of 3×3 and a fully connected layer are added. Finally, a three-dimensional feature vector of size H×W×D is output, namely: feature map F; where H, W, and D are the height, width, and number of channels of feature map F, respectively.

4. The multi-scale recognition method for handwritten mathematical formulas based on a counting module according to claim 1 is characterized in that: The channel attention module consists of a global average pooling layer, two linear layers and an activation function layer; The preliminary symbol features are given different channel weights after the global average pooling layer, two linear layers and the activation function to obtain enhanced symbol features.

5. The multi-scale recognition method for handwritten mathematical formulas based on a counting module according to claim 1 is characterized in that: In step 33, the specific working process of the LSTM network is: Step B1: Use the forget gate in the LSTM network to calculate the context vector C of the previous time step t-1 t-1 Perform selective forgetting and obtain the forget gate output f t ; Step B2: The hidden state h of the previous time step t-1 is calculated through the input gate of the LSTM network t-1 and the input feature vector X Dt Perform feature screening to obtain the output i of the input gate t and Step B3: Output f of the forget gate obtained in step B1 t and the output i of the input gate obtained in step B2 t and Calculate the hidden state h output by the output gate of the LSTM network at the current time step t t and the context vector C t ; Step B4: Calculate the context vector C of the current time step t in the channel dimension t , hidden state h t Concatenate with the count vector V to predict the predicted sequence symbol at the current time step t And the corresponding symbol prediction probability p t ; Step B5: When the predicted sequence symbol After predicting the end symbol, concatenate the predicted sequence symbols of all time steps in order to obtain the final output sequence 6. The multi-scale recognition method for handwritten mathematical formulas based on a counting module according to claim 5 is characterized in that: In step B4, the following formula is used: p t =g([C t ;h t ;V]); Where g(·) is a linear transformation function followed by a softmax activation function, which is used to output the symbol prediction probability of the current time step t, and argmax is the symbol prediction probability p t Output the corresponding symbol.

Images