Mathematical formula recognition method and device based on coding and decoding, and readable storage medium
Through the Resnet network and multi-head attention model based on the codec model, the feature sequence of handwritten mathematical formulas is extracted and processed, and the complexity and timing dependence problems of character cutting and structure analysis in traditional handwritten mathematical formula recognition technology are solved, achieving high efficiency recognition effect.
Patent Information
- Application Number
- CN202111581067.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-22
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-12-22
AI Technical Summary
Traditional handwritten mathematical formula recognition technology has complexity and timing dependence in character cutting, recognition and structure parsing, resulting in poor results. Especially in offline environments, traditional optical character recognition technology is difficult to apply because of the inability to obtain stroke information.
Using a method based on the codec model, the feature sequence is extracted using the Resnet network and position coding, and parallel calculations are performed through the multi-head attention model to generate prediction sequences, reducing the complexity of serial computing time, and realizing the recognition of handwritten mathematical formulas.
The process of handwritten mathematical formula recognition is simplified, the model is run time-consuming, the recognition efficiency and accuracy are improved, and the distinction ability between categories is enhanced through angle constraints and modular length constraints.
Smart Images

Figure CN114255379B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image recognition, and in particular to an end-to-end offline handwritten mathematical formula recognition method and device based on a coding and decoding model. Background Art
[0002] Mathematical formulas are often used in daily life, and are even more common in education, scientific and technological work, and other scenarios. Therefore, the effective recognition of mathematical formulas has become a very important task. Among them, handwritten mathematical formulas are more practical because of their convenience. However, unlike ordinary text, mathematical formulas often contain complex two-dimensional structures, and in offline handwritten mathematical formulas, traditional optical character recognition technology is often not used because of the irregularity of handwritten characters and the inability to obtain stroke information. Therefore, handwritten mathematical formula recognition is not only an important task in the current handwriting recognition field, but also a challenging task.
[0003] In the field of traditional image recognition technology, offline handwritten mathematical formula recognition is mainly based on image segmentation, character recognition and semantic analysis, such as projection-based segmentation technology, contour mapping method, recognition technology based on Convolutional Neural Network (CNN), grammar-based structural analysis method and structural pattern recognition method, etc. However, due to the changes in characters themselves, adhesion between characters, complex structure, differences in personal handwriting habits, etc., the application of traditional methods in handwritten mathematical formula recognition is relatively complex and the effect is not good.
[0004] In recent years, with the rapid development of artificial intelligence technology, handwritten mathematical formula recognition methods based on deep learning have emerged. These methods often do not require separate character cutting, recognition and structural analysis. For example, a convolutional neural network (CNN) is used to extract and encode image features, and a recurrent neural network (RNN) is used to decode and output. These methods are usually based on the recurrent neural network structure, and have problems with time dependence and computational complexity. Summary of the invention
[0005] To solve the above technical problems, the present invention provides a mathematical formula recognition method and device based on encoding and decoding, and a readable storage medium. It does not require explicit character cutting, character recognition and structure analysis. Instead, it adopts a Resnet network and position encoding to extract and calculate feature sequences, and then performs parallel calculations on the feature sequences based on a multi-head attention model to generate a prediction sequence, and finally completes the recognition of the entire handwritten mathematical formula, reducing the serial calculation time complexity in general formula recognition.
[0006] The mathematical formula recognition method based on coding and decoding described in the present invention comprises the following steps:
[0007] Step 1: Select a picture containing only handwritten mathematical formulas as a sample, and crop and grayscale the input picture;
[0008] Step 2: Input the adjusted image into the encoding network to obtain a feature sequence;
[0009] Step 3: Decode image features and predict characters through a decoding network;
[0010] Step 4: Calculate the loss between the character prediction result and the correct value through the L-softmax loss function to evaluate the prediction result.
[0011] Furthermore, in step 1, the image including all the formula parts is cropped to 256*256 pixels.
[0012] Furthermore, in step 2, the encoding network is formed by a fusion of an improved ResNet convolutional network and position coding, wherein the improved ResNet convolutional network adopts ResNet18, including a convolutional layer and a linear network, and the convolutional layer outputs a feature sequence of 512×8×8. After being processed by the linear network, a feature sequence x with a dimension of 210×8×8 is obtained; then the feature sequence x is calculated by the position coding module, and the absolute position information of each sequence is attached to obtain the final feature sequence F, which is the input sequence of the decoding network.
[0013] Furthermore, the position encoding module is as follows:
[0014]
[0015]
[0016]
[0017]
[0018] Among them, pos is the position of the current sequence, and the value range is 0 to d len -1,d len is the preset formula length, which is 210 in this embodiment; i is the dimension position in the current sequence, and its value range is 0 to d model -1,d model is the feature sequence dimension size.
[0019] Furthermore, the decoding network consists of 4 sub-networks and a fully connected network. Each sub-network contains a multi-head self-attention network and a forward network. The input of each layer of the sub-network is a 210×8×8 sequence, in which the multi-head self-attention network performs parallel operations on the relationship between the sequences and outputs the results to the forward network. The forward network is passed to the sub-network of the next layer after the same dimension conversion. The final result of the decoding network is calculated by the fully connected network, with a dimension of 210×d voc , where d voc The length of the set to which the predicted character belongs.
[0020] Furthermore, the multi-head attention model expression is:
[0021] MultiHead(Q,K,V)=Concat(head1,...,head h )W°,
[0022]
[0023] Among them, Q, K, and V are the feature sequences input to the multi-head attention. is a trainable parameter in the multi-head attention module, d model is the feature sequence dimension, d k =d model / h is the numerical scaling factor.
[0024] Furthermore, the L-softmax loss function formula is as follows:
[0025]
[0026] Where N is the number of training data, i represents the i-th sequence currently generated, and w yi is the weight parameter of the last fully connected layer, x i is the i-th output sequence, θ i Represents the angle between the current fully connected layer and the current sequence; ψ(θ) is the angle constraint function used by the L-softmax function:
[0027]
[0028] where D(θ) is any monotonically decreasing function, and equal m is an integer greater than 1 that is closely related to the classification boundary.
[0029] A device applied to the mathematical formula recognition method based on coding and decoding, the device comprising:
[0030] Image processing module, used to crop and grayscale the pictures containing only formulas;
[0031] A feature coding module connected to the image processing module is used to extract image feature information and calculate and add position information by position coding;
[0032] The feature decoding module connected to the feature encoding module is used for image feature sequence decoding and character prediction.
[0033] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above method.
[0034] The beneficial effects described in the present invention are as follows: the present invention uses a ResNet network to perform feature extraction in the encoding stage and combines position encoding to calculate the position information of the feature, thereby enhancing the feature extraction effect; uses L-softmax to update the network parameters during training, and enhances the ability to distinguish between categories through angle constraints and module length constraints; adopts a method of inputting the feature sequence as a whole into the self-attention model for parallel calculation, thereby improving the temporal dependency problem of the sequence prediction of the model prediction result, effectively simplifying the network structure, and reducing the running time of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.
[0036] Figure 1 This is a flow chart of offline handwritten mathematical formula recognition in a specific implementation manner of the present invention;
[0037] Figure 2 This is a multi-head attention module in the offline handwritten mathematical formula recognition network structure diagram in a specific implementation manner of the present invention. DETAILED DESCRIPTION
[0038] Embodiment 1:
[0039] The mathematical formula recognition method based on encoding and decoding described in the present invention is as follows: Figure 1 As shown, the specific steps include:
[0040] Step 1: Image preprocessing: crop and grayscale the image containing only handwritten mathematical formulas. The image only contains all the formula parts and as little blank area as possible, and the size is cropped to 256*256 pixels.
[0041] Step 2: Image feature encoding: The processed image is passed through the encoding network formed by the fusion of the improved ResNet convolutional network and the position encoding module to obtain the input of the decoding network.
[0042] Traditional convolutional neural networks face the problem of gradient vanishing / exploding as the network depth increases. Therefore, ResNet introduces a residual network structure, which introduces a forward feedback shortcut connection between the input and output so that the gradient during training can be directly transferred in the opposite direction.
[0043] As shown in Table 1 below, the improved ResNet convolutional network in the present invention adopts ResNet18. Unlike the original ResNet18 network, the last fully connected layer is replaced with a linear network. The final network output of the convolutional layer of the improved ResNet convolutional network is a 512×8×8 feature sequence. After being processed by conv6, a feature sequence x is obtained, and its dimension is 210×8×8. The parameters of each layer in the ResNet18 network are all trainable parameters.
[0044]
[0045] Table 1
[0046] Then, through linear transformation, the dimension of the feature sequence x is adjusted to 210×128 and sent to the position encoding module to calculate the position information between the feature sequences x and itself. The calculation formula is:
[0047] F=x+Pe(x)
[0048]
[0049]
[0050]
[0051]
[0052] Among them, pos is the position of the current sequence, and the value range is 0 to d len -1,d len is the preset formula length, which is 210 in this embodiment; i is the dimension position in the current sequence, and its value range is 0 to d model -1,d model is the feature sequence dimension size, which is 128 in this embodiment. According to the knowledge of trigonometric functions, for any offset k, PE pos+k Can be made by PE pos Linear representation, so the position-encoded sequence F can express the position information between feature sequences.
[0053] Step 3: The decoding network consists of 4 sub-networks and a fully connected network. Each sub-network contains a multi-head self-attention network and a forward network. The input of each layer of the sub-network is a 210×8×8 sequence. The multi-head self-attention network performs parallel operations on the relationship between the sequences and outputs the results to the forward network. The forward network is converted to the same dimension and then passed to the sub-network of the next layer. The final result of the decoding network is calculated by the fully connected network, with a dimension of 210×d voc , where d voc The length of the set to which the predicted character belongs.
[0054] The expression of the multi-head self-attention network in the sub-network is:
[0055] MultiHead(Q',K',V')=Concat(head1,...,head h )W°,
[0056] in, Among them, Q, K, and V are the feature sequences input to the multi-head attention.
[0057] is a trainable parameter in the multi-head attention module, d model is the feature sequence dimension, d k =d model / h is a numerical scaling factor; because Q, K, and V in the method are all equal to the decoding network input F, this network is called a self-attention network. The output dimension of the self-attention network is d len ×d model , after linear transformation, the output is d len ×d dict The predicted sequence (d dict is the length of the set of characters to be predicted including the start, end and blank tags), select d len The maximum prediction result in the sequence of each dimension in is the recognition result of the handwritten mathematical formula.
[0058] During the training phase, the L-softmax function is used to impose constraints on the relationship between prediction sequences. The loss between the character prediction results and the correct values is calculated through the L-softmax loss function. The prediction results are evaluated to guide the update of the parameters of each layer of the ResNet18 network and the trainable parameters in the multi-head attention model. The value obtained after the prediction output is subjected to 1softmax operation represents the similarity between the predicted value and the correct value. The smaller the value, the more similar they are.
[0059] The L-softmax loss function formula is as follows:
[0060]
[0061] Where N is the number of training data, i represents the i-th sequence currently generated, and w yi is the weight parameter of the last fully connected layer, x i is the i-th output sequence, θ i Represents the angle between the current fully connected layer and the current sequence; ψ(θ) is the angle constraint function used by the L-softmax function:
[0062]
[0063] where D(θ) is any monotonically decreasing function, and equal m is an integer greater than 1 that is closely related to the classification boundary; ψ(θ) selected in the present invention is:
[0064] Where m=3.
[0065] The loss function adopted by the present invention adds an angle constraint and a modulus length constraint to the standard Softmax loss to ensure that the boundaries between samples of different categories are more obvious.
[0066] This experiment was run on a server equipped with an Intel(R) Core(TM) i9-10900X CPU@3.70FHz and two NVIDIA TX2080Ti graphics cards, and the operating system was Linux Ubuntu 18.04. The deep learning framework was Pytorch 1.4.0, and the main required libraries were python3.7.9 and torchvision.
[0067] The training and validation datasets for the experiments of the present invention are HAMEX, which is 4350 handwritten mathematical expressions written by 58 authors, of which 90% are training sets and 10% are validation sets.
[0068] The specific implementation algorithm is to first use the HAMEX dataset for 50 iterative training, and the sample size of each batch is set to 8; the method of neural network parameter adjustment adopts the Adam optimization method, in which the learning rate is set to 0.001, and the smoothing constant is set to 0.9 and 0.98; the Adam optimization method is combined with the L-softmax loss function to perform parameter training and update, and the loss function is minimized by changing the parameters in the neural network.
[0069] Embodiment 2:
[0070] A device applied to the mathematical formula recognition method based on coding and decoding, the device comprising:
[0071] Image processing module, used to crop and grayscale the pictures containing only formulas;
[0072] A feature coding module connected to the module is used to extract image feature information and calculate and add position information by position coding;
[0073] The feature decoding module connected to the module is used for image feature sequence decoding and character prediction. The encoding network is composed of sub-networks, each of which contains a multi-head self-attention network and a forward network; the decoding network is used to calculate the positional relationship of the feature sequence and output a predictive sequence. The L-softmax function is used to impose stronger constraints on the relationship between the prediction sequences. Through these prediction sequences, the best character path is selected, and each character finally recognized is output in sequence.
[0074] The various variations and specific examples of the mathematical formula recognition method based on encoding and decoding in the aforementioned embodiment 1 are also applicable to an offline handwritten mathematical formula recognition device based on an encoding and decoding model in this embodiment. Through the aforementioned detailed description of an offline handwritten mathematical formula recognition method based on an encoding and decoding model, those skilled in the art can clearly know the implementation system of an end-to-end offline handwritten mathematical formula recognition device in this embodiment, so for the sake of brevity of the specification, it will not be described in detail here.
[0075] Embodiment three:
[0076] Based on the offline handwritten mathematical formula recognition method based on the encoding and decoding model in the aforementioned embodiment 1, the present invention further provides a computer-readable storage medium, which stores a computer program, and when the program is executed by a processor, the steps of the above method are implemented.
[0077] The specific examples described herein are merely examples of the spirit of the present invention. A person skilled in the art of the present invention may make various modifications or additions to the specific examples described or replace them in a similar manner, but this will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.
[0078] Those skilled in the art will appreciate that embodiments of the present invention may be provided as systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0079] The present invention should understand that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that has the functions specified in one or more boxes.
[0080] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0082] The above description is only a preferred embodiment of the present invention and is not intended to be a further limitation of the present invention. All equivalent changes made using the contents of the present specification and drawings are within the protection scope of the present invention.
Claims
1. A mathematical formula recognition method based on encoding and decoding, characterized in that: The steps of the method are as follows: Step 1: Select a picture containing only handwritten mathematical formulas as a sample, and crop and grayscale the input picture; Step 2: Input the adjusted image into the encoding network to obtain a feature sequence; The encoding network is formed by the fusion of an improved ResNet convolutional network and position coding, wherein the improved ResNet convolutional network adopts ResNet18, including a convolutional layer and a linear network, the convolutional layer outputs a 512×8×8 feature sequence, and after being processed by the linear network, a feature sequence x with a dimension of 210×8×8 is obtained; then the feature sequence x is calculated by a position coding module, and the absolute position information of each sequence is added to obtain the final feature sequence F, that is, the input sequence of the decoding network; Step 3: Decode image features and predict characters through a decoding network; The decoding network consists of 4 sub-networks and a fully connected network. Each sub-network contains a multi-head self-attention network and a forward network. The input of each layer of the sub-network is a 210×8×8 sequence. The multi-head self-attention network performs parallel operations on the relationship between the sequences and outputs the results to the forward network. The forward network is passed to the sub-network of the next layer after the same dimension conversion. The final result of the decoding network is calculated by the fully connected network with a dimension of 210×d voc , where d voc The length of the set to which the predicted character belongs; Step 4: Calculate the loss between the character prediction result and the correct value through the L-softmax loss function to evaluate the prediction result.
2. The method for mathematical formula recognition based on encoding and decoding according to claim 1, characterized in that: In step 1, the image including all the formula parts is cropped to 256*256 pixels.
3. The method for mathematical formula recognition based on encoding and decoding according to claim 1, characterized in that: The position encoding module is as follows: Among them, pos is the position of the current sequence, and the value range is 0 to d len -1,d len is the preset formula length, i is the dimension position in the current sequence, and the value range is 0 to d model -1,d model is the feature sequence dimension size.
4. The method for mathematical formula recognition based on coding and decoding according to claim 1, characterized in that: The multi-head attention model expression is: MultiHead(Q,K,V)=Concat(head1,...,head h )W°, Among them, Q, K, V are the feature sequences input to the multi-head attention, and W o ∈d model ×d model , is a trainable parameter in the multi-head attention module, d model is the feature sequence dimension, d k =d model / h is the numerical scaling factor.
5. The method for mathematical formula recognition based on encoding and decoding according to claim 1, characterized in that: The L-softmax loss function formula is as follows: Where N is the number of training data, i represents the i-th sequence currently generated, and w yi is the weight parameter of the last fully connected layer, x i is the i-th output sequence, θ i Represents the angle between the current fully connected layer and the current sequence; ψ(θ) is the angle constraint function used by the L-softmax function: where D(θ) is any monotonically decreasing function, and equal m is an integer greater than 1 that is closely related to the classification boundary.
6. A device applied to the mathematical formula recognition method based on coding and decoding as described in any one of claims 1 to 5, characterized in that: The device comprises: Image processing module, used to crop and grayscale images containing only formulas; A feature coding module connected to the image processing module is used to extract image feature information and calculate and add position information by position coding; The feature decoding module connected to the feature encoding module is used for image feature sequence decoding and character prediction.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.
