A method for printed mathematical formula recognition based on character-level features

Abstract

The present invention discloses a method for recognizing printed mathematical formulas based on character-level features, which is characterized by adopting an end-to-end neural network architecture, using an unsupervised image segmentation algorithm to extract character-level features of mathematical formulas, and using a position-aware attention mechanism to construct character-level positional relationships, so that the model can capture the complex positional relationship between characters in a two-dimensional space. The method specifically includes the steps of character-level image feature extraction, character-level positional relationship extraction, LaTeX sequence decoding, and model training. Compared with the prior art, the present invention has the advantages of simple method, high accuracy, no need to accurately segment characters, no need to determine what characters the segmented image blocks are, no error accumulation, greatly improving the accuracy of the model in the task of recognizing printed mathematical formulas, and having good and broad application prospects.

Description

Technical Field

[0001] The invention relates to the technical field of computer text recognition, in particular to a method for recognizing printed mathematical formulas based on character-level features. Background Art

[0002] Mathematical formula understanding is widely used in many smart education scenarios, including automatic problem solving, personalized exercise recommendation, and student learning assessment. Since mathematical formulas are usually saved in the form of printed images, it is not conducive to editing, searching, semantic understanding, and other operations on the formulas. Therefore, converting printed images of mathematical formulas into structured expressions (such as LaTeX, character layout trees) has become a very important basic task. This process is called printed mathematical formula recognition. Compared with optical character recognition (OCR), printed mathematical formula recognition not only needs to identify all the characters in the image, but also needs to obtain the spatial relationship between the characters in the image.

[0003] The traditional method of printed mathematical formula recognition includes three stages: character segmentation, character recognition, and layout analysis. This not only requires a large number of manually defined rules, but also causes error accumulation between different tasks, and the recognition accuracy is not high. With the development of deep learning, the encoder-decoder model is used to solve the problem of image-to-text generation. Many methods used for picture description problems are used to complete formula recognition tasks. Although the encoder-decoder model improves the performance of printed mathematical formula recognition to a certain extent compared with traditional methods, the following problems still exist:

[0004] 1) Compared with the picture description task, the LaTeX sequence length of the printed mathematical formula recognition output is longer. In MS COCO, a common dataset for the picture description task, the average length of the text description of the picture is 10.47 characters. In contrast, the average length of the LaTeX sequence of mathematical formulas appearing in academic papers is 62.78 characters. Cho et al. proposed that the performance of the encoder-decoder model for text generation will decrease rapidly as the sentence length increases. Therefore, the picture description model cannot solve the printed formula recognition problem well.

[0005] 2) Compared with the picture description task, the recognition of printed mathematical formulas requires capturing more precise position information. In the picture description task, a small change in the position of an object in the picture sometimes does not affect the semantics of the sentence. The picture description model is not sensitive to slight changes in the position of an object. In the picture of printed mathematical formulas, slight changes in the position of characters will lead to changes in the mathematical semantics. This requires the picture description model to have the ability to capture the precise position relationship of characters in order to solve the problem of printed mathematical formula recognition.

[0006] 3) Printed math formula recognition is a complete description of all character information in the image. It not only needs to recognize all Roman letters, Greek letters, operators and other characters in the image, but also needs to recognize the spatial relationship between all characters to generate a complete LaTeX sequence. As long as there is an error in the sequence, the entire math formula recognition process will fail. The purpose of picture description is to describe the general idea of ​​the image, and there is no need to show all the details in the image. Therefore, the picture description task model is not suitable for direct use in printed math formula recognition. Summary of the invention

[0007] The purpose of the present invention is to design a printed mathematical formula recognition method based on character-level features in view of the deficiencies in the prior art. The present invention adopts an end-to-end neural network architecture, uses an unsupervised image segmentation algorithm to extract character-level features of mathematical formulas, combines the position information between characters on the basis of the character-level features, and uses a position-aware attention mechanism to construct character-level position relationships, so that the model can capture the precise position information between characters, perform complex relationship extraction operations in two-dimensional space and output long-distance text sequences. There is no need to segment the characters completely and accurately, nor is there any need to determine what characters the segmented image blocks are, and no error accumulation will occur. The method is simple, greatly improves the accuracy of the model in the task of printed mathematical formula recognition, and has good and broad application prospects.

[0008] The object of the present invention is achieved as follows: a method for recognizing printed mathematical formulas based on character-level features, which is characterized by adopting an end-to-end neural network architecture, using an unsupervised image segmentation algorithm to extract character-level features of mathematical formulas, and using a position-aware attention mechanism to construct character-level positional relationships, so that the model can capture the complex positional relationship between characters in a two-dimensional space. The recognition of printed mathematical formulas specifically includes the following steps:

[0009] 1) Character-level image feature extraction: Use an unsupervised image segmentation algorithm to segment the printed mathematical formula image to obtain multiple character-level image blocks; use a neural network model to extract features from each image block to obtain all character-level image features of the printed mathematical formula.

[0010] 2) Character-level position relationship extraction: Calculate the normalized coordinate vector of each image block in the original image, use a neural network model to encode the coordinate vector of each image block to obtain all character-level position features of the printed mathematical formula, combine the character-level image features and the character-level position features, and use a position-aware encoder module to extract the position relationship between image blocks to obtain the character-level features of the printed mathematical formula image.

[0011] 3) LaTeX sequence decoding: Use a decoder that can capture long-distance text information to decode the character-level features of the printed mathematical formula image obtained in the previous step and output the LaTeX sequence of the image.

[0012] 4) Model training: Use unsupervised image segmentation algorithms with different thresholds to perform data augmentation on the images. Define a loss function and use an optimizer to train the model until the loss function converges.

[0013] In the character-level image feature extraction step, for the segmentation of the printed mathematical formula image, a connected domain segmentation algorithm is used, a threshold is set to divide the image into different image blocks, and each image block is scaled proportionally and padded to the same square size. For extracting the features of each image block, a convolutional neural network is used to extract and generate a character-level feature vector of the image block.

[0014] In the character-level position relationship extraction step, for the calculation of the normalized position coordinate vector of the image block, the distance between each image block and the top, bottom, left and right boundaries in the original printed mathematical formula image is calculated and normalized, and the aspect ratio of the original image is added to the position coordinate vector. For extracting the position relationship between image blocks, the transformer model is used as the basic model, and the position-aware attention mechanism is used to replace the original attention mechanism in the transformer.

[0015] In the LaTeX sequence decoding step, for decoding the character-level features of the printed mathematical formula image, a transformer model that can capture long-distance dependencies in the sequence is used as a decoder.

[0016] In the model training step, the cross entropy loss function is used as the training loss, and the label smoothing method is used to prevent the neural network model from overfitting. In the training stage, different thresholds are set for the connected domain segmentation algorithm to obtain different segmentation combinations of the complete image, and then the Adam optimizer is used to optimize the model until the model loss function converges.

[0017] The present invention adopts an end-to-end neural network architecture and uses an unsupervised image segmentation algorithm to segment images. It not only obtains the character-level features of printed mathematical formula images, but also enhances the data input by selecting multiple thresholds of unsupervised segmentation algorithms, and has fine-grained feature extraction capabilities. Based on the transformer model, the position-aware attention mechanism is used, so that the transformer model has the ability to capture character-level position relationships, greatly improving the accuracy of printed mathematical formula recognition.

[0018] Compared with the prior art, the present invention has the advantages of simplicity and high accuracy. It does not require completely accurate segmentation of characters, nor does it require judgment of what characters the segmented image blocks are. It does not produce error accumulation, greatly improving the accuracy of the model in the task of printed mathematical formula recognition, and has good and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of printed mathematical formula recognition in Example 1. DETAILED DESCRIPTION

[0020] The present invention uses an unsupervised image segmentation algorithm to extract character-level features of mathematical formulas. Based on the character-level features, the position information between characters is combined, and the position-aware attention mechanism is used to construct the character-level position relationship, so that the model can capture the precise position information between characters. The Transformer model as a position encoder and decoder enables the model to perform complex relationship extraction operations in two-dimensional space and output long-distance text sequences.

[0021] The present invention will be further described in detail in conjunction with the following specific embodiments and drawings. Except for the contents specifically mentioned below, the processes, conditions, experimental methods, etc. for implementing the present invention are common knowledge and common common sense in the field and are not particularly limited by the present invention.

[0022] Example 1

[0023] See also Figure 1 The present invention recognizes printed mathematical formulas according to the following steps:

[0024] 1) Character-level image feature extraction: Use the connected domain segmentation algorithm to segment the entire printed mathematical formula image into character-level image blocks, preprocess each image block, and use a convolutional neural network to extract the character-level feature vector of each image block. The specific steps are as follows:

[0025] Step 1: Use the connected domain segmentation algorithm, set the segmentation threshold t, and segment the printed mathematical formula image according to the threshold t to obtain the image block set S = {s1, s2, ..., s n}.

[0026] Step 2: Transform each image block s in S i The longest side (height or width) of is scaled to length b, and then padded to a b*b square image to make all image blocks of the same size.

[0027] Step 3: Use convolutional neural network to extract features from all image blocks and obtain character-level image features i for each image block. i ,i iis an m-dimensional vector.

[0028] 2) Character-level position relationship extraction: First, the normalized coordinate vector of each image block in the original image is calculated, and then the coordinate vector of each image block is encoded using a fully connected network to obtain character-level position features. The character-level image features and character-level position features are then combined, and the position relationship between images is extracted using a position-aware encoder module to obtain character-level features of the printed mathematical formula image. The specific steps are as follows:

[0029] Step 4: Calculate the position feature p of each image block in the original printed mathematical formula image i , p i ={t i / H,d i / H,l i / W,r i / W,H / W}. Among them, t i ,d i ,l i ,r i They are image blocks s i The distance from the top, bottom, left, and right boundaries of the original image; H and W are the height and width of the original image.

[0030] Step 5: Use a fully connected neural network to train p i Encode to obtain the character-level position feature p of each image block i ', the character-level position feature p i is an m-dimensional vector.

[0031] Step 6: Transform character-level image features i i and character-level position feature p i Add together to get the character-level feature vector e i At this point, the original printed mathematical formula image has been represented as n m-dimensional vectors, whose vector set is E = {e1, e2, ..., e n}.

[0032] Step 7: Use a transformer model to encode the vectors in set E, and use the position-aware attention mechanism to replace the original attention mechanism in the transformer model. By calculating the attention score of each element in set E, we finally get the encoded representation set R = {r1, r2, ..., r n}, the position-aware attention mechanism is expressed by the following equations (a) to (c):

[0033]

[0034]

[0035]

[0036] 3) LaTeX sequence decoding: Use a transformer model as a decoder to predict the LaTeX sequence. The specific steps are as follows:

[0037] Step 8: <start>As the initial character, the m-dimensional absolute position code input at the previous moment is calculated by trigonometric function coding, and the coding matrix is ​​used to map the character at the previous moment in the LaTeX sequence into an m-dimensional semantic vector.

[0038] Step 9: Add the m-dimensional absolute position encoding and the m-dimensional semantic vector to obtain the semantic representation of the previous time step as the input of transformer decoding.

[0039] Step 10: Take the vector in the R set obtained in step 7 as the output result of the encoder and input it into the transformer decoding model. Calculate the attention score with the semantic representation of the previous moment in step 9, and finally get the LaTeX character prediction of the current time step.

[0040] 4) Model training: Use different thresholds of connected domain segmentation algorithms to segment the original printed mathematical formula image, obtain multiple groups of image blocks as data enhancement within the model, define a loss function, and use the Adam optimizer to train the model until the model loss function converges. The specific steps are as follows:

[0041] Step 11: Use different connected domain segmentation algorithm thresholds to segment the original image, turning one group of samples into multiple groups of samples, achieving the effect of data enhancement.

[0042] Step 12: Use the cross entropy loss function as the final loss of the model and the label smoothing method as the regularization method.

[0043] Step 13: Train the model using the Adam optimizer until the model loss function converges.

[0044] Compared with the prior art picture-speaking model, the present invention has the following significant technical advances and advantages:

[0045] 1) The present invention introduces a character segmentation module into the deep learning method, and provides character-level fine-grained image features for the deep learning picture-speaking model. This method is different from the character segmentation module in the prior art in that the present invention uses an unsupervised image cutting algorithm, which does not require completely accurate segmentation of characters, nor does it require judgment of what characters the segmented image blocks are, and does not cause error accumulation between stages.

[0046] 2) This paper uses fine-grained character-level image features and introduces character-level position features. Combined with the position-aware attention mechanism, the model can capture the complex positional relationships between characters in two-dimensional space.

[0047] 3) The present invention has conducted a large number of experiments on the Im2Markup dataset, and the experimental results show that it is superior to the existing multi-stage methods and deep learning methods in terms of accuracy.

[0048] The above is only a further explanation of the present invention and is not intended to limit this patent. Any equivalent implementation of the present invention should be included in the scope of the claims of this patent.< / start>

Claims

1. A method for recognizing printed mathematical formulas based on character-level features, characterized in that An end-to-end neural network architecture is adopted, and the connected domain segmentation algorithm is used to extract the character-level features of mathematical formulas. The position-aware attention mechanism is used to construct the character-level position relationship, so that the model can capture the complex position relationship between characters in two-dimensional space. The recognition of printed mathematical formulas specifically includes the following steps: (I) Character-level image feature extraction Use the connected domain segmentation algorithm to segment the printed mathematical formula image to obtain multiple character-level image blocks, and use the neural network model to extract features from each image block to obtain all character-level image features of the printed mathematical formula; (II) Character-level position relationship extraction Calculate the normalized coordinate vector of the position of each image block in the original image, use a neural network model to encode the coordinate vector of each image block to obtain all character-level position features of the printed mathematical formula, combine the character-level image features and the character-level position features, and use a position-aware encoder module to extract the position relationship between the image blocks to obtain the character-level features of the printed mathematical formula image. The position-aware encoder module uses a transformer model as a basic model and uses a position-aware attention mechanism to replace the original attention mechanism in the transformer. (III) LaTeX sequence decoding Use a decoder that can capture long-distance text information to decode the character-level features of the printed math formula image obtained in the previous step and output the LaTeX sequence of the image; (IV) Model training Use unsupervised image segmentation algorithms with different thresholds to perform data augmentation on the images, define a loss function, and use an optimizer to train the neural network model until the loss function converges; The position-aware attention mechanism is expressed by the following equations (a) to (c):

2. The method for recognizing printed mathematical formulas based on character-level features according to claim 1, characterized in that In the character-level image feature extraction step, for the segmentation of the printed mathematical formula image, a connected domain segmentation algorithm is used, a threshold is set to divide the image into different image blocks, and each image block is scaled proportionally and padded to the same square size. The features of each image block are extracted using a convolutional neural network to generate a character-level feature vector of the image block.

3. The method for recognizing printed mathematical formulas based on character-level features according to claim 1, characterized in that In the character-level position relationship extraction step, the position coordinate vector of the image block is normalized, the distance between each image block and the top, bottom, left and right boundaries in the original printed mathematical formula image is calculated and normalized, and the aspect ratio of the original image is added to the position coordinate vector. For extracting the position relationship between image blocks, the transformer model is used as the basic model, and the position-aware attention mechanism is used to replace the original attention mechanism in the transformer.

4. The method for recognizing printed mathematical formulas based on character-level features according to claim 1, characterized in that In the LaTeX sequence decoding step, for decoding the character-level features of the printed mathematical formula image, a transformer model that can capture long-distance dependencies in the sequence is used as a decoder.

5. The method for recognizing printed mathematical formulas based on character-level features according to claim 1, characterized in that In the model training step, a cross entropy loss function is used as the training loss, and a label smoothing method is adopted to prevent overfitting of the neural network model. In the training stage, different thresholds are set for the connected domain segmentation algorithm to obtain different segmentation combinations of the complete picture, and the Adam optimizer is used to optimize the neural network model until the model loss function converges.

Images