A Chinese environment mathematical formula extraction and recognition method based on a Transformer

Abstract

The application discloses a Chinese environment mathematical formula extraction and recognition method based on a Transformer. The method comprises the following steps: inputting a picture containing a mathematical formula into a mathematical formula mask extraction module in a Chinese environment, and extracting a mask containing the mathematical formula, wherein only the mathematical formula part is reserved and the text part at other positions is removed in the mask; inputting the mask into a mathematical formula picture extraction module, and further repairing and perfecting the mathematical formula position in the mask through a morphological method, and generating a mathematical formula picture according to the repaired mask; inputting the mask into a conversion display module, and obtaining the Latex of the formula through the module, and generating a mathematical formula picture through Katex to display and compare. The method improves the accuracy of formula extraction and formula recognition based on the Transformer technology. Unlike the prior art, the application can automatically infer the formula region of the input picture, and is more convenient to use.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and natural language processing, in particular to a Chinese environment mathematical formula extraction and recognition method based on Transformer, which is a method combining computer vision and Transformer in natural language processing, combining network structure and Transformer. The accuracy of formula extraction and recognition is improved, and the extracted content and results are visualized through a visual interface. BACKGROUND

[0002] With the continuous development of Internet technology, people's office needs are also constantly improving. Online office has become the norm for people, and the most widely used in online office is PDF documents. PDF is a document format developed by Adobe for publishing and storing electronic information, which is globally used. It can ensure accurate and accurate printing results on any printer. In order to achieve this effect, PDF documents contain a large amount of rendering information. This also makes the original content save in different forms due to the PDF format, such as a formula in a PDF document may be stored in text form, or it may be saved as an image form. Such a saving form is convenient for us to print, but in our learning and life, we will always encounter the reuse of some formulas in PDF, such as saving the formulas in the document into notes or saving the formulas in the document into papers. At this time, the saving form of the formula in the PDF will make it very difficult to reuse, so people often have to manually type it again.

[0003] Because of people's needs, there are many excellent developers who have developed many formula extraction software and websites, such as the "Formula King" website and the mathpix software. They can achieve good recognition results, but at the same time, the user needs to manually segment the formula area and the segmentation must be accurate. If there is text in the formula area, it will greatly affect the recognition result. The existing technology provides image to Latex conversion, but does not complete the automatic extraction of formulas in the image. The present application realizes the automatic extraction of mathematical formulas in Chinese environment and the conversion of images to Latex. In the present application, the input can be a whole picture containing mathematical formulas, and the mathematical formula extraction model will automatically filter out the text information and give the formula position. The user only needs to fine-tune the area to get the mathematical formula Latex character in that area. SUMMARY

[0004] The present application provides a Chinese environment mathematical formula extraction and recognition method based on the prior art of Transformer. Through this method, a whole picture containing mathematical formula is input, the mathematical formula extraction model automatically filters out the text information and gives the mathematical formula position, and after manual fine-tuning, the Latex character can be input into the recognition model to finally obtain the Latex character. Compared with the previous recognition software, the present application does not need to manually accurately frame, the software can automatically give the formula area, and the user only needs to fine-tune, which can more conveniently use.

[0005] The technical scheme adopted by the present application is:

[0006] A Chinese environment mathematical formula extraction and recognition method based on the Transformer, comprising the following steps:

[0007] 1) inputting a picture containing mathematical formula in a Chinese environment into a mathematical formula mask extraction module, obtaining a mask containing mathematical formula through the extraction of the module, and only retaining the mathematical formula part in the mask and removing the text part at other positions;

[0008] 2) inputting the mask containing mathematical formula obtained after step 1) into a mathematical formula picture extraction module, further repairing and perfecting the mathematical formula position in the mask through a morphological method, and finally generating a mathematical formula picture according to the repaired mask;

[0009] 3) inputting the mathematical formula picture processed in step 2) into a conversion display module, obtaining the Latex of the formula through the module, and generating a mathematical formula picture through Katex to display and compare.

[0010] Further, the step 1) is:

[0011] 1.1) inputting a group of Chinese environment pictures containing mathematical formula X as input, ensuring that the size of each picture is 512*512 through data strengthening, and the picture is a gray image, and the channel number of input is 1;

[0012] 1.2) constructing a mathematical formula extraction model, the model structure is divided into two parts, one part is a convolution downsampling part, which extracts high-dimensional features of the picture through downsampling, and the other part is to reconstruct the picture based on the extracted high-dimensional features; guided by the loss function, the model can filter out the text information, but retain the formula;

[0013] 1.3) Down-sampling input image X into several convolution layers, each of which is composed of convolution calculation, activation function, Batch Normalization normalization, and the feature map obtained by passing through each convolution layer is a high-dimensional feature vector representing the characteristics of the picture. After passing through a series of convolution layers, a feature map with a size of 1x1 and a channel number n is obtained;

[0014] 1.4) Convert the high-dimensional feature vector obtained in 1.3) into a two-dimensional matrix with dimensions HxWxC into (HxW)x C, that is, obtain a 1xn matrix. Take the n channel values as a sequence with a length of n, and input the sequence into the natural language processing model Transformer for encoding and further feature extraction;

[0015] 1.5) The Transformer first performs position encoding on the input sequence, and then performs multi-head attention mechanism calculation on the obtained data. The results obtained by each attention mechanism are spliced to obtain the final result. Then, the input and output of the multi-head attention mechanism are added and normalized by LayerNorm, and then input into the feedforward neural network. The result is added to the input of the feedforward neural network and normalized by LayerNorm;

[0016] 1.6) The operation of 1.5) is a Block module in the Transformer encoder. After passing through a plurality of Block modules in turn, the features are further extracted to obtain a 1xn feature matrix. Then, the feature matrix is converted into a high-dimensional feature vector with a size of 1x1xn;

[0017] 1.7) Up-sampling reconstruction, input the high-dimensional feature data obtained from 1.6) and the feature maps obtained by each convolution layer during down-sampling into the deconvolution layer. The number of deconvolution layers is the same as that of convolution layers and corresponds layer by layer. The structure is composed of transposed convolution, activation function, and Batch Normalization normalization. Before inputting each deconvolution layer, the down-sampled feature map obtained by the corresponding convolution layer is spliced and input through skip connection. Finally, the feature matrix 1x1xn is restored to a 512x512 grayscale image through up-sampling;

[0018] 1.8) After the calculation of the model from 1.1) to 1.7), a mask containing the position of the mathematical formula is needed. A loss function composed of Tversky Loss and improved focal loss is used

[0019]

[0020] Wherein TP refers to the number of pixels in the position predicted by the model and judged by the model to be the real mathematical formula, FN refers to the number of pixels in the position predicted by the model but not judged by the model to be the real mathematical formula, FP refers to the number of pixels in the position predicted by the model but not judged by the model to be the real mathematical formula, IOU refers to the intersection of the area of the mathematical formula predicted by the model and the area of the real mathematical formula divided by the union; α, β, γ are hyperparameters;

[0021] 1.9) After the model is trained according to the loss function of 1.8), a mask with the same size as the original Chinese picture containing mathematical formula can be obtained by calculation of the model after inputting the Chinese picture containing mathematical formula, and only the mathematical formula part is retained in the mask and the text part at other positions is removed. Figure One

[0022] Further, the step 2) is:

[0023] 2.1) After obtaining the mask containing mathematical formula of the image by step 1), the opencv algorithm is used to extract each formula from the mask for input of the recognition module, and the obtained mask image is first subjected to morphological erosion calculation to make the mask connected regions through;

[0024] 2.2) After the morphological processing of the image, the pixels in the image are projected in the vertical direction, and a threshold Vε is set. When the distance between the continuous projections is less than Vε, it is considered that it is part of the same formula, and the merging between the projections is performed. When the distance between the projections is greater than Vε, it is considered that it is a different formula, and the distance of the projection, i.e. the vertical pixel segment of the image, is stored in the array Vproj.

[0025] 2.3) After obtaining the vertical pixel segment array Vproj, the image is subjected to vertical direction segmentation. According to each vertical pixel segment, the image is divided into size(Vproj) images, and then each vertically segmented image is subjected to horizontal direction projection. A threshold Hε is set. When the distance between the continuous projections is less than Hε, it is considered that it is part of the same formula, and the merging between the projections is performed. When the distance between the projections is greater than Hε, it is considered that it is a different formula, and the distance of the projection, i.e. the horizontal pixel segment of the image, is stored in the array Hproj. According to the horizontal pixel segment, the image can be segmented again on the basis of the vertical segmentation to obtain the image of each formula and stored.

[0026] Further, the step 3) is:

[0027] ​3.1) Constructing the conversion display model, the conversion to Latex deep learning model is mainly composed of two parts, respectively, the encoder and the decoder, the encoder adopts the backbone network is ResnetV2 Vision Transformer, the decoder is decoded by the Transformer decoder to get the Latex finally.

[0028] 3.2) A set of mathematical formula pictures generated in step 2) as input, first input these pictures into ResnetV2 as the backbone network, after the down sampling of the backbone network, the picture is divided into 16x16 parts, so after the backbone network we get the feature map size

[0029] 3.3) The feature map obtained by 3.2) is converted into a two-dimensional matrix, that is, a matrix, the feature map is regarded as a sequence with a length of , and the Embedding of each word is a 256 vector, and the sequence is input into the natural language processing model Transformer for coding to further extract features;

[0030] 3.4) The input sequence is positionally encoded, and the obtained data is calculated by multi-head attention mechanism, and the results obtained by each attention mechanism are spliced to obtain the results, and the input and output of the multi-head attention mechanism are added and normalized by LayerNorm, and then input into the feedforward neural network, and the results are added to the input of the feedforward neural network and normalized by LayerNorm.

[0031] 3.5) The operation of 3.4) is a Block module in the Transformer encoder, which successively passes through multiple Block modules, and finally outputs a context global matrix Enc with the same size , the encoding stage is completed.

[0032] 3.6) is the encoding stage from 3.2) ~ 3.5), and then the feature values are decoded, the decoder is composed of the decoder of the Transformer, the seq sequence of the formula field and the context global matrix Enc are input into the decoder of the Transformer, the decoder also encodes the input sequence first, then inputs the masked multi-head attention layer, then adds the input and output of the masked multi-head attention layer and performs LayerNorm normalization, and then inputs the data into the multi-head attention layer after the calculation of the masked multi-head attention layer, but in this layer, the input of the attention mechanism K matrix and V matrix is replaced by the context global matrix Enc, then the input and output of the multi-head attention layer are added and LayerNorm normalized, and then input into the feedforward neural network, the input and output of the feedforward neural network are added and LayerNorm normalized to get the final value;

[0033] 3.7) 3.6) is a block module of the Transformer decoder, and multiple block modules are connected to complete decoding; after calculation, a matrix Dec of seq_len x num_tokens size is obtained, where seq_len represents the length of the field input into the model, and num_tokens is a hyperparameter that can be set, which represents the maximum possible number of words;

[0034] 3.8) the loss function is

[0035] L = CE (Decoder (x = seq, : :-1, context = Encoder (img)) T , seq[:,1:])

[0036] Where img represents the input formula picture, Encoder represents the VisionTransformer with ResNetV2 as the backbone network, seq represents the corresponding token sequence of the formula Latex, Decoder represents the decoder of the Transformer, and CE represents the cross-entropy formula; In the loss function, the input formula field seq is divided into two parts, seq, :-1) is excluded from the last character as input, and seq, :1) is excluded from the first character as output target;

[0037] 3.9) After training the model through the loss function of 3.8), input the picture, and 4 outputs the Latex predicted according to the picture;

[0038] 3.10) The generated Latex is converted into a picture through the Katex module in Nodejs to visualize the display of the Latex formula.

[0039] The beneficial effects of this invention are:

[0040] This invention constructs a mathematical formula mask extraction model, trained on a large set of images containing Chinese characters. When such images are input into the model, it can filter out the textual information while retaining the formula information, resulting in a mask that is identical to the original. Figure One A mask of a specific size is used. After processing, the mask is input into a mathematical formula image extraction module. In this module, the formula region is extracted from the original image using the OpenCV algorithm and the information provided by the mask and saved locally. A transformation and display model is trained based on a large number of mathematical formula images and LaTeX character information. When the extracted formula image is input into this model, the model automatically infers the LaTeX characters. This invention combines Transformer technology from natural language processing, enabling accurate extraction and recognition of mathematical formulas in a Chinese environment. Compared to existing technologies, this invention can automatically infer and recognize the formula region of the input image, making it more convenient to use. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the process of this invention.

[0042] Figure 2 This is a mathematical formula mask extraction model diagram of the present invention.

[0043] Figure 3 This is a flowchart of the mathematical formula image extraction module of the present invention.

[0044] Figure 4 This is a transformation demonstration model diagram of the present invention.

[0045] Figure 5 This is a screenshot of an example of the mathematical formula mask extraction module of the present invention.

[0046] Figure 6 This is a screenshot of an example of the mathematical formula image extraction module of this invention.

[0047] Figure 7 This is a screenshot of an example of the interactive module of the present invention.

[0048] Figure 8 This is a screenshot of an example of the conversion display module of the present invention. Detailed Implementation

[0049] The present invention will be further described in detail below with reference to the accompanying drawings, and the purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0050] Referring to Figure 1 The flowchart between the various modules in the present application can be seen.

[0051] A Chinese environment mathematical formula extraction and recognition method based on Transformer, comprising the following steps:

[0052] 1) Mathematical formula mask extraction: by establishing a segmentation model and training a labeled data set, when a picture containing a mathematical formula in a Chinese environment is input into the model, the position mask containing the mathematical formula can be obtained through the extraction of the model, and other characters are removed;

[0053] 2) Mathematical formula picture extraction: the mathematical formula mask obtained after step 1) is input into this module, which further repairs and perfects the mathematical formula position in the mask through morphological methods, and finally generates mathematical formula pictures according to the repaired mask.

[0054] 3) Conversion and display: by establishing a recognition model and training a labeled data set, when a mathematical formula picture is input into the model, the Latex of the formula can be obtained through the calculation of the model. And through Katex, the mathematical formula picture can be generated again for display and comparison.

[0055] Specifically, referring to Figures 2-8 , it is illustrated in combination with examples:

[0056] A Chinese environment mathematical formula extraction and recognition method based on Transformer, wherein the mathematical formula mask extraction module can filter out text information and only keep the formula, and the model structure is as shown in Figure 2 . The step 1) is specifically:

[0057] First, get the mathematical formula picture in the Chinese environment, read the training samples through the opencv function library, first invert the picture to black and white, and reduce the picture to 1 / 4 of the original picture. At the same time, the picture is processed by morphological dilation. And stack several original Figure 1 / 4 pictures, and then crop to get a 512x512 size picture. Because the neural network calculation cannot bear large pictures, such data enhancement method can make the information contained in the training sample more rich, and the morphological dilation processing also prevents some thin formulas from being ignored during training. The model graph is as shown in Figure 2 .

[0058] 1.1) Input a set of Chinese environment pictures containing mathematical formulas X, and ensure that the size of each picture is 512x512 and that it is a gray image. The input channel number is 1.

[0059] 1.2) Convolution on input image X, kernel size 4x4, padding 1, stride 2, output channel n. After this convolution, the image size is reduced by half to 256x256, channel number n. n is a hyperparameter that can be specified by yourself. This step is called encoder 1.

[0060] 1.3) LeakyReLU activation with parameter 0.2 on the input data of 1.2), and then do a convolution with kernel size 4x4, padding 1, stride 2, output channel 2n, and then do a BatchNormalization. The feature map size becomes 128x128, channel number 2n. This step is called encoder 2.

[0061] 1.4) LeakyReLU activation with parameter 0.2 on the input data of 1.3), and then do a convolution with kernel size 4x4, padding 1, stride 2, output channel 2n, and then do a BatchNormalization. The feature map becomes 64x64, channel number 2n. This step is called encoder 3.

[0062] 1.5) Repeat the steps of 1.3) and 1.4) to perform deeper convolution on the image, but the output channel number becomes 4n. The feature map size obtained by encoder 4 is 32x32, channel number 4n. The feature map size obtained by encoder 5 is 16x16, channel number 4n.

[0063] 1.6) Repeat the steps of 1.3) and 1.4) to perform deeper convolution on the image, but the output channel number becomes 8n. The feature map size obtained by encoder 6 is 8x8, channel number 8n. The feature map size obtained by encoder 7 is 4x4, channel number 8n. Figure 4

[0064] 1.7) Repeat the steps of 1.3) and 1.4) to perform deeper convolution on the image, but the output channel number becomes 16n. The feature map size obtained by encoder 8 is 2x2, channel number 16n. The feature map size obtained by encoder 9 is 1x1, channel number 16n. Figure 1

[0065] 1.8) Convert the data HxWxC obtained in 1.7) into a two-dimensional matrix of (HxW)xC, i.e. obtain a 1x16n matrix. We regard the 16n channel values as a sequence of length 16n, and input this sequence into the natural language processing model Transformer for encoding and further feature extraction.

[0066] ​​1.9)The Transformer encodes the input sequence first, and then the obtained data is calculated by multi-head attention mechanism. The results obtained by each attention mechanism are spliced to obtain the final result. Then the input and output of the multi-head attention mechanism are added and normalized by LayerNorm, and then input into the feedforward neural network. The result is added to the input of the feedforward neural network and normalized by LayerNorm.

[0067] 1.10)The operation of 1.9) is a Block in the Transformer encoder. The last encoder layer is composed of multiple Transformer encoder Blocks. After the Transform calculation, the obtained data is converted into the matrix shape before inputting into the Transformer by Reshape, that is, the size is 1x1x16n.

[0068] 1.11)The next step is to perform the decoder operation. The data obtained from 1.10) is input into the relu activation function, and then input into the transpose convolution with a size of 2x2 and a step of 2. After inputting into BatchNormalization for normalization, this step is called decoder8

[0069] 1.12)After decoding, we connect the feature maps of the decoder and the encoder through the skip connection of the network, which helps to reconstruct the image. Before decoder7 calculation, we splice the data obtained by decoder8 with the calculation result obtained by encoder8 to obtain the feature Figure One input into decoder7, which performs the same decoding steps as 1.11).

[0070] 1.13)Repeat the steps of 1.11) and 1.12) to continuously fuse features and upsample. Finally, a gray image with the same size as the original Figure One image of 512x512 and a channel number of 1 is obtained.

[0071] 1.14)1.1) to 1.13) are the contents of the model, and a mask containing mathematical formula positions needs to be obtained after the model calculation. Therefore, according to the loss function

[0072]

[0073] The TP in the loss function refers to the number of pixels in the predicted position that are judged correctly by the model and are real mathematical formula pixels, the FN refers to the number of pixels in the predicted position that are judged incorrectly by the model but are real mathematical formula pixels, and the FP refers to the number of pixels in the predicted position that are judged incorrectly by the model but are not real mathematical formula pixels. The IOU refers to the intersection of the area of the mathematical formula predicted by the model and the area of the real mathematical formula divided by the union. In the present application, the loss function is reduced, and the smaller the loss, the better the effect. The loss function is composed of two parts, the first half is Tversky Loss, which can adjust the proportion of false positive samples and false negative samples in the loss function through α, β, so as to achieve better results. The second half is the improved focal loss, which regards the IOU value as the accuracy of sample recognition, and adjusts the information amount of each sample through (1-IOU) 2 When the IOU is high, the information amount of the sample is greatly reduced, and when the IOU is low, the information amount of the sample is basically unchanged. Thus, the model training is more focused on difficult samples.

[0074] 1.15) After training the model according to the loss function of 1.14), a mask with the same size as the original Figure One image can be obtained by calculating the model after inputting a Chinese picture containing a mathematical formula. The results are shown in Figure 6 .

[0075] After obtaining the mask, the opencv algorithm can extract each formula from the mask. The flow chart is shown in Figure 3 .

[0076] The step 2) is specifically:

[0077] 2.1) After obtaining the mask of the image mathematical formula from 1.15), according to 1 / 4 size of the original image, the originally stacked mask is cut and overlapped, and then the mask image is obtained. The mask is first subjected to morphological erosion calculation to make the mask connected regions through. After morphological calculation, the area of each connected region is calculated and small area connected regions are removed. After morphological processing, the white region pixels of the mask on the original image are retained based on the mask, and the black region pixels of the mask on the original image are removed. Finally, an image containing only the formula is obtained.

[0078] 2.2) After getting the formula image, project the pixels in it to the vertical direction. Since there is a space between two formulas, the text in the space is removed. Set a threshold Vε, if the distance between two consecutive projections is less than Vε, it is considered as the same formula, and the projections are merged. If the distance between two consecutive projections is greater than Vε, it is considered as different formulas, and the distance of the projections, i.e. the vertical pixel segment of the image, is stored in the array Vproj.

[0079] 2.3) After getting the vertical pixel segment array Vproj, the image is segmented in the vertical direction. According to each vertical pixel segment, the image is divided into size(Vproj) images, and then each vertically segmented image is projected in the horizontal direction. Set a threshold Hε, if the distance between two consecutive projections is less than Hε, it is considered as the same formula, and the projections are merged. If the distance between two consecutive projections is greater than Hε, it is considered as different formulas, and the distance of the projections, i.e. the horizontal pixel segment of the image, is stored in the array Hproj. According to the horizontal pixel segment, the image can be segmented again based on the vertical segmentation to obtain the image of each formula and stored. The example effect diagram is shown in Figure 7

[0080] The threshold Vε for vertical segmentation can be set smaller, and in the present application, it can be set to 20. The threshold Hε for horizontal segmentation can be set to 40 considering various situations. Due to the complexity of horizontal segmentation, this step can be manually adjusted after this step to modify, delete or add the generated mathematical formula picture.

[0081] In the conversion display module, the picture can be converted into Latex characters and displayed, and the Latex can be further converted into a formula picture for comparison with the original picture. The model structure of the formula recognition is shown in Figure 5

[0082] 3.1) After manual adjustment, the picture can be converted into Latex. The deep learning model for converting into Latex mainly consists of two parts, an encoder and a decoder. The encoder adopts a Vision Transformer with a backbone of ResnetV2, and the decoder decodes by a normal Transformer decoder to finally obtain the Latex.

[0083] ​​3.2)Input into a set of pictures containing mathematical formulas X, the width of the picture W is set between 96 and 672, and the height of the picture H is set between 32 and 192. First, input these pictures into the backbone of ResnetV2. The processing of the backbone can divide the picture into 16x16 parts, so after the backbone, we should get the picture size

[0084] 3.3)ResnetV2 for Vision Transformer consists of 3 parts. First, the first part is that the picture X input in this module is first subjected to a convolution operation with a channel number of 64, a convolution kernel of 7x7, a step of 2, and a convolution method of SAME. The SAME convolution makes the image size after convolution become half of the original size according to the convolution kernel size, and the image size after convolution becomes the size, and the channel number is 64. Then the obtained feature map is subjected to Group Normalization, and then a convolution kernel of 3x3, a step of 2, and a maximum pooling calculation with an expansion of 1 are performed, so that the obtained feature map becomes the size, and the channel number is 64.

[0085] 3.4) After 3.3), the next step is to perform residual calculation of ResnetV2. This part can set hyperparameters, and in this invention, we set it to [3, 4, 9]. That is, the residual module consists of 3 small residual groups, each of which has 3, 4, and 9 residual blocks respectively. The first residual group is not down-sampled, and the second and third residual blocks are both down-sampled by 1 / 2. Finally, a feature map with a channel number of 1024 is obtained;

[0086] 3.5) After the processing of 3.4), the next step is to normalize the obtained feature map. Then perform adaptive pooling to ensure that the feature map size remains unchanged, and finally perform convolution on the feature map with a convolution kernel of 1 and a step of 1. The output channel number is a hyperparameter that can be set, and here it is set to 256. The obtained feature map is .

[0087] 3.6) Convert the data HxWxC obtained after processing the feature map by the backbone into a two-dimensional matrix (HxW)x C, that is, obtain the matrix. The feature map is regarded as a length of ​The sequence is given, where the embedding of each word is a 256-bit vector. This sequence is then input into the Transformer natural language processing model for encoding and further feature extraction.

[0088] 3.7) The input sequence is positionally encoded, and the resulting data is then processed using a multi-head attention mechanism. Multiple attention mechanisms are calculated individually, and their results are concatenated to obtain the final result. The input and output of the multi-head attention mechanism are then summed and normalized using LayerNorm, before being input into a feedforward neural network. The result is then summed with the input of the feedforward neural network and normalized again using LayerNorm.

[0089] The operations in 3.8) and 3.7) are a single block in Transformer encoding. A Transformer consists of multiple blocks, and the final output of each encoded Transformer Block is of the same size. The context global matrix Enc. The encoding phase is now complete.

[0090] 3.9) From 3.2) to 3.8) is the encoding stage. Next, the feature values ​​are decoded. The decoder consists of the Transformer's decoder. The seq sequence of the formula field and the context matrix Enc are input into the Transformer's decoder. The decoder also first performs positional encoding on the input sequence, but the input is fed into a masked multi-head attention layer. Unlike a regular multi-head attention layer, the mask obscures future information. That is, during decoding, the character y at position i... i Only the first i-1 characters will affect y. i The input and output of the attention layer are added together and then normalized using LayerNorm. After the masked multi-head attention layer is computed, the data is input into a regular multi-head attention layer. However, in this layer, the Q matrix input of the attention mechanism is still a general input, while the inputs of the K and V matrices are replaced with Enc with context information. The input and output of the attention layer are added together and then normalized using LayerNorm. The data is then input into the feedforward neural network, and the input and output of the feedforward neural network are added together and normalized using LayerNorm to obtain the final value.

[0091] 3.10) 3.9) is called a block of the Transformer decoder, and multiple blocks are connected to complete the decoding. Finally, a seq_len x num_tokens matrix Dec is obtained, where seq_len represents the length of the input field of the model, and num_tokens is a hyperparameter that can be set, representing the maximum possible number of tokens.

[0092] 3.11) The loss function is

[0093] L = CE (Decoder (x = seq, : :- 1, context = Encoder (img)) T , seq[:,1:])

[0094] where img represents the input formula picture, Encoder represents the VisionTransformer with ResNetV2 as the backbone. seq represents the corresponding token sequence of the formula Latex, Decoder represents the decoder of the Transformer, and CE represents the cross-entropy formula. In the loss function, the input formula field seq is divided into two parts, seq, :- 1 is excluded from the last character as input, and seq, : 1 is excluded from the first character as output target. The decoding matrix obtained in 3.10) is transposed into a num_tokens x (seq_len-1) matrix, and the cross-entropy calculation is performed with the target seq, : 1. The calculation result is used as its loss function. It is equivalent to predicting the sequence (1, seq_len) at the next position of the window (0, seq_len-1) given the sequence.

[0095] 3.12) After training the model with the loss function in 3.11), input the picture, and set seq to an array with only one 1, where 1 represents the start symbol and 2 represents the end symbol, i.e. [1] represents the start of generation. The matrix output by the model only takes the last item, which is the latest character prediction with a length of num_tokens. This item represents the prediction of the next Latex character based on the existing seq and picture input. The values of the top several tokens are retained, and the remaining values are all set to -inf. Then, the 1 x num_tokens matrix is normalized by softmax to obtain the probability of each character. The next item is predicted by random selection.

[0096] 3.13) After this prediction is completed, seq is lengthened to become [1, p1], and this is input into the model again as seq to repeat the operation of 3.12). Until the end symbol 2 appears or the maximum length is reached. The program ends. The output is the Latex obtained according to the picture prediction.

[0097] 3.14) The generated Latex is converted into a picture by the Katex module in Nodejs to visualize the formula. The correctness of the generated Latex can be judged by directly comparing the generated formula picture with the original formula picture. An example is shown in Figure 8 .

[0098] The method of the present application can realize accurate automatic extraction of mathematical formulas in a Chinese environment, and can provide users with mathematical formula Latex characters in the region without operation or only with fine adjustment of the region, so that the use is more convenient.

Claims

1. A Transformer-based Chinese environment mathematical formula extraction and recognition method, characterized by comprising the following steps: 1) Input the picture containing mathematical formula in Chinese environment into the mathematical formula mask extraction module, and obtain the mask containing mathematical formula through the extraction of the module, wherein only the mathematical formula part is retained and the text part at other positions is removed in the mask; 2) Input the mask containing mathematical formula obtained after step 1) into the mathematical formula picture extraction module, the module further repairs and perfects the mathematical formula position in the mask through morphological method, and finally generates the mathematical formula picture according to the repaired mask; 3) Input the mathematical formula picture processed in step 2) into the conversion display module, obtain the Latex of the formula through the module processing, and generate the mathematical formula picture through Katex to display and compare; The step 1) is: 1.1) A set of Chinese environment pictures containing mathematical formulas As input, data enhancement is used to ensure that the size of each picture is 512 512, and is a grayscale image, the number of channels of the input is 1; 1.2) Construct a mathematical formula extraction model, the model structure is divided into two parts, one part is the convolution downsampling part, through the downsampling to extract the high-dimensional features of the picture, the other part is to reconstruct the picture based on the extracted high-dimensional features; guided by the loss function, this model can filter out the text information, but also retain the formula; 1.3) down-sampling the input image Input into several convolutional layers, each of which is composed of three steps of convolution calculation, activation function, Batch Normalization normalization. The feature map obtained after passing through each convolutional layer is a high-dimensional feature vector that can represent the characteristics of the picture. After passing through a series of convolutional layers, a feature map with a size of 1 1 and a channel number of 512 is finally obtained. ; 1.4) Convert the high-dimensional feature vector obtained in 1.3) into a two-dimensional matrix, with dimensions Convert into , i.e., obtain a matrix of , and take this channel value as a sequence of length , and input this sequence into the natural language processing model Transformer for encoding and further feature extraction; 1.5) The Transformer first encodes the input sequence to obtain data, then calculates the multi-head attention mechanism, separately calculates multiple attentions, then splices the results obtained by each attention mechanism to obtain the result; then add the input and output of the multi-head attention mechanism and perform LayerNorm normalization, then input into the feedforward neural network, get the result and add the input of the feedforward neural network and perform LayerNorm normalization; 1.6) the operation of 1.5) is a Block module in the Transformer encoder, and the features are further extracted through a plurality of Block modules in turn to obtain the feature matrix of 1 , and then the feature matrix is converted into a high-dimensional feature vector, i.e. 1 1 ​ 1.7) upsampling reconstruction, input the high-dimensional feature data obtained from 1.6) and the feature maps obtained from each convolution layer during downsampling into the deconvolution layer, the number of deconvolution layers is the same as that of the convolution layers and each layer corresponds to each other, the structure is composed of transpose convolution, activation function and Batch Normalization normalization; before inputting each deconvolution layer, the down-sampling feature maps obtained by the corresponding convolution layer are spliced and input through the skip connection, and finally the feature matrix is restored to a 512 1 gray image through upsampling; 512 1.8) After the calculation of the models in 1.1) to 1.7), the mask containing the position of the mathematical formula is needed, which is obtained by using the loss function composed of TverskyLoss and improved focal loss ； Wherein TP refers to the number of pixel points in the position predicted by the model and judged correctly by the model and which are real mathematical formulas, FN refers to the number of pixel points in the position predicted by the model and which are not real mathematical formulas, FP refers to the number of pixel points in the position predicted by the model and which are not real mathematical formulas, and IOU refers to the intersection of the area of the mathematical formula predicted by the model and the area of the real mathematical formula divided by the union; 、 , γ is a hyperparameter; 1.9) After training the model according to the loss function in 1.8), inputting a Chinese picture containing a mathematical formula, the mask with the same size as the original picture can be obtained through model calculation, and only the mathematical formula part is retained in the mask while the text part at other positions is removed.

2. The Chinese environment mathematical formula extraction and recognition method based on the Transformer according to claim 1, characterized in that, The step 2) is: 2.1) After obtaining the mask containing the mathematical formula of the image through step 1), the opencv algorithm is used to extract each formula from the mask for input into the recognition module, and first the obtained mask image is subjected to morphological erosion calculation to make the mask connected regions through; 2.2) After morphological processing, the image is projected in the vertical direction, and a threshold is set If the distance between two successive projections is less than then it is considered as part of the same formula, and the merging between projections is performed, if the distance between two successive projections is greater than then it is considered as a different formula, and the distance between projections, i.e. the vertical pixel segment of the image, is stored in an array ; 2.3) get vertical pixel segment array After that, the image is divided vertically, and the image is divided into images according to each vertical pixel segment, and then each vertical divided image is projected horizontally, and a threshold is set When the distance between the continuous projections is less than , it is considered to be part of the same formula, and the merging between the projections is performed, and when the distance between the projections is greater than , it is considered to be a different formula, and the distance of the projection, i.e. the horizontal pixel segment of the image, is stored in an array , and according to the horizontal pixel segment, the image of each formula can be divided again on the basis of the vertical division, and stored.

3. The Chinese environment mathematical formula extraction and recognition method based on the Transformer according to claim 1, characterized in that, The step 3) is: 3.1) Construct a conversion display model, the deep learning model converted to Latex is mainly composed of two parts, namely the encoder and the decoder, the encoder adopts the Vision Transformer composed of ResnetV2 backbone network, and the decoder is decoded by the Transformer decoder to finally obtain Latex; 3.2) Take the set of mathematical formula pictures generated in step 2) as input, first input these pictures into ResnetV2 as the backbone network, and after downsampling of the backbone network, divide the pictures into 16 16 parts, so after the backbone network, the feature map size we get is ; 3.3) convert the feature map obtained in 3.2) into a two-dimensional matrix, i.e. a matrix of size , where the Embedding of each word is a vector of size 256, and input this sequence into the natural language processing model Transformer for encoding and further feature extraction; ​ 3.4) The input sequence is positionally encoded, and the obtained data is then calculated by a multi-head attention mechanism, with multiple attentions being calculated separately, then the results obtained by each attention mechanism are spliced to obtain the result, and the input and output of the multi-head attention mechanism are added and LayerNorm normalized, then input into the feedforward neural network, get the result and add the input of the feedforward neural network and perform LayerNorm normalization; 3.5) the operation of (3.4) is a Block module in the Transformer encoder, sequentially passing through multiple Block modules, and finally outputting the same size the context global matrix , the encoding stage is completed; 3.6) From 3.2) ~ 3.5) is the encoding stage, and then the feature value is decoded, and the decoder is composed of the decoder of the Transformer, the seq sequence of the formula field, and the context global matrix The input into the decoder of the Transformer, the decoder also encodes the input sequence first, inputs the masked multi-head attention layer, then adds the input and output of the masked multi-head attention layer and performs LayerNorm normalization, and then inputs the data into the multi-head attention layer after the calculation of the masked multi-head attention layer, but in this layer, the input of the attention mechanism K matrix and V matrix are all using the context global matrix Instead, then add the input and output of the multi-head attention layer and perform LayerNorm normalization, and then input into the feedforward neural network, add the input and output of the feedforward neural network and perform LayerNorm normalization to get the final value; 3.7) 3.6) is a block module of the Transformer decoder, and multiple block modules are needed to complete the decoding. After calculation, the final matrix of size where seq len refers to the field length input into the model, and num tokens is a settable hyperparameter referring to the maximum possible number of words; 3.8) The loss function is ； Where img represents the input formula picture, Encoder represents the VisionTransformer whose backbone network is ResNetV2, seq refers to the corresponding token sequence of the formula Latex, Decoder refers to the decoder of the Transformer, and CE refers to the cross-entropy formula; in the loss function, the input formula field seq is divided into two parts, seq[:,:-1] is excluded from the last character as input, and seq[:,1:] is excluded from the first character as output target; 3.9) After training the model by the loss function of 3.8), input the picture, and output the Latex obtained by predicting according to the picture; 3.10) The generated Latex is converted into a picture by the Katex module in Nodejs to visualize the display of the Latex formula.

4. A Chinese environment mathematical formula extraction and recognition system based on a Transformer, characterized in that, It includes: A mathematical formula mask extraction module for extracting the position mask of mathematical formulas in pictures containing mathematical formulas in a Chinese environment and removing other text information; A mathematical formula picture extraction module for morphological processing of the output of the mathematical formula mask extraction module to further repair and improve the position of the mathematical formula in the mask and generate a mathematical formula picture; A conversion and display module for calculating the Latex of the formula from the output of the mathematical formula picture extraction module and generating a mathematical formula picture for display and comparison through Katex.

5. The Chinese environment mathematical formula extraction and recognition system based on the Transformer according to claim 4, characterized in that, It also includes an interactive module for visualizing the mathematical formula picture generated by the mathematical formula picture extraction module, and according to the user's settings, the formula picture can be modified, added or deleted, and after the operation, the previous mathematical formula picture output is replaced.

Images