A syntax-directed end-to-end trainable online handwritten mathematical formula recognition method

Abstract

The application provides a syntax-guided end-to-end trainable online handwritten mathematical formula recognition method, which comprises the following steps: generating a database for training a model and training the model; and using the trained model to analyze a handwritten mathematical formula. The application solves the problem that a handwritten mathematical formula is difficult to be accurately recognized by combining a prediction analysis method for analyzing LL(1) syntax with a bundle search to complete a mathematical formula decoding process, through synthesizing a large number of online handwritten mathematical formula samples from online handwritten mathematical formula samples and mathematical formula samples.

Description

Technical Field

[0001] This invention relates to the field of pattern recognition technology, specifically to a syntax-guided, end-to-end trainable online handwritten mathematical formula recognition method. Background Technology

[0002] Online handwritten mathematical formula recognition is a technique that converts the dynamic handwriting of handwritten mathematical formulas into a structured format (such as MathML). Since mathematical formulas are an important information carrier commonly found in various documents, and with the widespread use of pen input devices, this technology has broad application prospects. Compared to natural language text recognition, mathematical formula recognition is more difficult due to its more complex and compact planar structure; compared to printed text recognition, handwriting recognition is also more difficult due to the need to handle varying writing habits; compared to offline recognition, online recognition is more likely to achieve higher accuracy due to additional temporal information and less background noise. Currently, two known methods can achieve the highest level of accuracy:

[0003] The first type of method has a clearly defined symbol recognition module and a structure analysis module. The former obtains a set of candidate symbols by segmenting and recognizing stroke sequences, while the latter reconstructs mathematical formulas by analyzing the positional relationships between symbols and parsing two-dimensional random context-free grammar. The two complement each other to complete the recognition of handwritten mathematical formulas. Companies that use this type of method include MyScript (formerly Vision Objects) and Samsung. The advantage of this type of method is that the intermediate results have better interpretability, which makes it easier to design interactive user interfaces that support various editing operations and to customize the recognition range after training. However, since this type of method makes more use of human prior knowledge, there are more detailed issues involved in implementing them.

[0004] The second type of method is based on encoder-decoder models with attention mechanisms. Companies that use this type of method include iFlytek. The advantage of this type of method is that it is easy to implement because the model is end-to-end trainable and does not have to rely on a large number of difficult-to-maintain heuristic rules. However, since the model is largely a black box, a common problem with this type of method in the past is the lack of customizability at runtime. In addition, this type of model generally requires a large amount of training data to achieve high accuracy.

[0005] Therefore, it is evident that establishing a flexible online handwritten mathematical formula recognition system based on a limited number of handwritten mathematical formula samples is a problem that needs to be solved. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention proposes a syntax-guided, end-to-end trainable online handwritten mathematical formula recognition method.

[0007] To achieve the above objectives, the specific solution of the present invention is as follows:

[0008] This invention provides a syntax-guided, end-to-end trainable online handwritten mathematical formula recognition method, comprising the following steps:

[0009] S1, synthesize the training set and train the model;

[0010] S2 uses a trained model to parse handwritten mathematical formulas;

[0011] Specifically, step S2 includes:

[0012] S21, Initialization state;

[0013] S211, initialize the set H of hypotheses consisting of the string representation of the formula, confidence level, decoder state, and predictive analysis stack as {(λ,1,h,(w0, <start>))}, where λ represents an empty string and h is the encoder output, <start>This is the start symbol for the syntax G;

[0014] S212, initialize the set Y of recognition candidates, which consists of the string representation of the formula and the confidence score, to an empty set;

[0015] S22, Repeat the following steps until the assumed set is empty:

[0016] S221, order

[0017]

[0018] The next segments are (w0, ..., w) n The confidence levels p0, ..., p) n There is also the updated state h′; this is calculated by the decoder based on the intermediate state h and the recognized string y.

[0019] S222, for each (y,p,h,t)∈H, repeat the following steps:

[0020] S2221, let t = (Z1, ..., Z) m ,X) while w is the last terminator of string y;

[0021] S2222, if X is a terminal symbol:

[0022] If X = w, then let t ← (Z1, ..., Z). m Otherwise, remove the current hypothesis from H and consider other hypotheses;

[0023] S2223, if X is not a terminal symbol:

[0024] If the predictive parsing table of LL(1) syntax G has a usable production "X←X1...X" when parsing a string starting with a terminal symbol w into a non-terminal symbol X. l ", then let t←(Z1, ..., Z m X l (, ..., X1); otherwise, remove the current hypothesis from H and consider other hypotheses;

[0025] S2224, if w = w0, then remove the current assumption from H and let Y←Y∪{(y, p)} and k←k-1;

[0026] S223, retain only the k hypotheses with the highest scores in H.

[0027] Furthermore, step S1 specifically includes:

[0028] S11 expands the online handwritten mathematical formula library;

[0029] S12, expands the structured mathematical formula library;

[0030] S13, For each structured mathematical formula, synthesize the corresponding handwritten handwriting;

[0031] S14, use the synthetic training set to train the model.

[0032] Furthermore, step S11 specifically includes:

[0033] S111, add all sub-formulas of each handwritten formula in the online handwritten mathematical formula library to the online handwritten mathematical formula library;

[0034] S112, extract the strokes of each mathematical formula in image form and add it to the online handwritten mathematical formula library.

[0035] Furthermore, step S12 specifically includes S121: generating a batch of mathematical formulas in MathML format according to random context-free syntax and adding them to the structured mathematical formula library.

[0036] Furthermore, step S13 specifically includes the following steps:

[0037] S131 renders the structured mathematical formulas into printed form, while noting the outer bounding box of each symbol.

[0038] S132, search for maximal sub-formulas with the same structure in the online handwritten mathematical formula library, and if found, adjust the position of the outer square of the symbol according to the handwritten sample.

[0039] S133, replace each symbol with one of the corresponding handwritten symbols in the online handwritten mathematical formula library, and then perform a slight random affine transformation;

[0040] S134, randomly rotates and scales the synthesized handwriting.

[0041] Furthermore, the database used to train the model includes: online handwritten mathematical formula samples, offline handwritten mathematical formula samples, printed mathematical formula samples, mathematical formula corpus, and randomly generated mathematical formulas.

[0042] Furthermore, training is based on data sources in the following forms:

[0043] A sample library of online handwritten mathematical formulas, where each online handwritten mathematical formula is saved in InkML format, and also includes annotations saved in MathML format;

[0044] A library of offline handwritten and printed mathematical formula samples, each of which includes an image saved in PNG format and annotations saved in MathML format;

[0045] A corpus of mathematical formulas, where each formula is saved in MathML format.

[0046] Furthermore, during the decoding process, it is ensured that the identified mathematical formulas have string representations that conform to the given syntax; the predictive analysis method used to parse LL(1) syntax is combined with beam search to complete the mathematical formula decoding process.

[0047] The technical solution of this invention has the following beneficial effects:

[0048] By synthesizing a large number of online handwritten mathematical formula samples from online handwritten mathematical formula samples and combining the predictive analysis method used to parse LL(1) syntax with beam search to complete the mathematical formula decoding process, the problem of the difficulty in accurately recognizing handwritten mathematical formulas is solved. Attached Figure Description

[0049] Figure 1 This is an overall flowchart of the present invention;

[0050] Figure 2 Flowchart for training the model to synthesize the training set;

[0051] Figure 3 To use a trained model to parse handwritten mathematical formula flowcharts. Detailed Implementation

[0052] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0053] In the description of this invention, unless otherwise explicitly specified and limited, the terms "connected," "linked," and "fixed" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0054] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0055] In the description of this embodiment, the terms "upper," "lower," "front," "rear," "left," and "right," etc., refer to the orientations or positional relationships shown in the accompanying drawings. They are used only for ease of description and simplification of operation, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention. Furthermore, the terms "first" and "second" are used merely for descriptive distinction and have no special meaning.

[0056] like Figure 1-3 As shown, this invention proposes a syntax-guided, end-to-end trainable online handwritten mathematical formula recognition method, comprising the following steps:

[0057] S1, synthesize the training set and train the model;

[0058] S2 uses a trained model to parse handwritten mathematical formulas;

[0059] Specifically, step S1 includes:

[0060] S11 expands the online handwritten mathematical formula library;

[0061] S111, add all sub-formulas of each handwritten formula in the online handwritten mathematical formula library to the online handwritten mathematical formula library;

[0062] S112, extract the strokes of each mathematical formula in image form and add it to the online handwritten mathematical formula library.

[0063] S12, expands the structured mathematical formula library;

[0064] S121: Generate a batch of mathematical formulas in MathML format according to random context-free syntax and add them to the structured mathematical formula library.

[0065] S13, For each structured mathematical formula, synthesize the corresponding handwritten handwriting;

[0066] S131 renders the structured mathematical formulas into printed form, while noting the outer bounding box of each symbol.

[0067] S132, search for maximal sub-formulas with the same structure in the online handwritten mathematical formula library, and if found, adjust the position of the outer square of the symbol according to the handwritten sample.

[0068] S133, replace each symbol with one of the corresponding handwritten symbols in the online handwritten mathematical formula library, and then perform a slight random affine transformation;

[0069] S134, randomly rotates and scales the synthesized handwriting.

[0070] S14, use the synthetic training set to train the model.

[0071] S2, using the trained model, parse out k handwritten mathematical formulas that conform to the given LL(1) syntax G and give the corresponding confidence scores.

[0072] S21, Initialization state;

[0073] S211, initialize the set H of hypotheses consisting of the string representation of the formula, confidence level, decoder state, and predictive analysis stack as {(λ,1,h,(w0, <start>))}, where λ represents an empty string and h is the encoder output, <start>w0 is the start symbol of syntax G, and w0 is the end symbol.

[0074] S212, initialize the set Y of recognition candidates, which consists of the string representation of the formula and the confidence score, to an empty set;

[0075] S22, Repeat the following steps until the assumed set is empty:

[0076] S221, order

[0077]

[0078] The next segments are (w0, ..., w) n The confidence levels p0, ..., p) n There is also the updated state h′; this is calculated by the decoder based on the intermediate state h and the recognized string y.

[0079] S222, for each (y,p,h,t)∈H, repeat the following steps:

[0080] S2221, let t = (Z1, ..., Z) m ,X) while w is the last terminator of string y;

[0081] S2222, if X is a terminal symbol:

[0082] If X = w, then let t ← (Z1, ..., Z). m Otherwise, remove the current hypothesis from H and consider other hypotheses;

[0083] S2223, if X is not a terminal symbol:

[0084] If the predictive parsing table of LL(1) syntax G has a usable production "X←X1...X" when parsing a string starting with a terminal symbol w into a non-terminal symbol X. l ", then let t←(Z1, ..., Z m X l (, ..., X1); otherwise, remove the current hypothesis from H and consider other hypotheses;

[0085] S2224, if w = w0, then remove the current assumption from H and let Y←Y∪{(y, p)} and k←k-1;

[0086] S223, retain only the k hypotheses with the highest scores in H.

[0087] The database used to train the model includes: online handwritten mathematical formula samples, offline handwritten mathematical formula samples, printed mathematical formula samples, mathematical formula corpus, and randomly generated mathematical formulas.

[0088] An online handwritten mathematical formula sample library, where each online handwritten mathematical formula is saved in InkML format, and also includes annotations saved in MathML format; an offline handwritten and printed mathematical formula sample library, where each offline handwritten mathematical formula is accompanied by an image saved in PNG format and annotations saved in MathML format; and a mathematical formula corpus, where each formula is saved in MathML format.

[0089] During the decoding process, ensure that the identified mathematical formulas have string representations that conform to the given syntax; combine the predictive analysis method used to parse LL(1) syntax with beam search to complete the mathematical formula decoding process.

[0090] The principle of this invention is as follows:

[0091] By adding a syntax-guided mechanism during the decoding process, the legality of the recognition results can be ensured, avoiding the generation of invalid LaTeX or MathML strings by the sequence-to-sequence model, such as cases where tags that should be paired are not paired or necessary sub-formulas are missing.

[0092] By using grammar to limit the range of recognition results, a model can be trained to be applicable to a variety of application scenarios. For example, a handwriting calculator may need to support the function "sin" but not allow the single symbol "s". In this case, grammar can be used to implement this restriction and distinguish highly similar symbols.

[0093] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made under the inventive concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of the present invention.< / start> < / start> < / start> < / start>

Claims

1. A syntax-directed end-to-end trainable online handwritten mathematical formula recognition method, characterized in that Includes the following steps: S1, synthesize the training set and train the model; S2 uses a trained model to parse handwritten mathematical formulas; Specifically, step S2 includes: S21, Initialization state; S211, a set of hypotheses consisting of string representations of formulas, confidences, decoder states, and prediction parse stacks initialized to where represents an empty string, is an output of the encoder, is a start symbol of the grammar , and is an end symbol; S212, initialize the set Y of recognition candidates, which consists of the string representation of the formula and the confidence score, to an empty set; S22, Repeat the following steps until the assumed set is empty: S221, order , The next segment is respectively confidence level And the updated status h '; is the decoder based on the intermediate state h and the identified strings y Calculated S222, for each Repeat the following steps: S2221, Note and For string The last terminal symbol; S2222, if For terminal symbols: like X = w Then let Otherwise, change the current assumption from Remove and reconsider other assumptions; S2223, if Non-terminal symbols: If LL(1) syntax In the predictive analysis table, strings starting with the terminal symbol 'w' are parsed as non-terminal symbols. Productions are available from time to time. "Then let Otherwise, change the current assumption from Remove and reconsider other assumptions; S2224, if Then change the current hypothesis from Remove and order and ; S223, only retained The highest score One hypothesis.

2. The syntax-guided end-to-end trainable online handwritten mathematical formula recognition method according to claim 1, characterized in that, Step S1 specifically includes: S11 expands the online handwritten mathematical formula library; S12, expands the structured mathematical formula library; S13, For each structured mathematical formula, synthesize the corresponding handwritten handwriting; S14, use the synthetic training set to train the model.

3. The syntax-guided end-to-end trainable online handwritten mathematical formula recognition method according to claim 2, characterized in that, Step S11 specifically includes: S111, add all sub-formulas of each handwritten formula in the online handwritten mathematical formula library to the online handwritten mathematical formula library; S112, extract the strokes of each mathematical formula in image form and add it to the online handwritten mathematical formula library.

4. The syntax-guided end-to-end trainable online handwritten mathematical formula recognition method according to claim 2, characterized in that, Step S12 specifically includes S121: generating a batch of mathematical formulas in MathML format according to random context-free syntax and adding them to the structured mathematical formula library.

5. The syntax-guided end-to-end trainable online handwritten mathematical formula recognition method according to claim 2, characterized in that, Step S13 specifically includes the following steps: S131 renders the structured mathematical formulas into printed form, while noting the outer bounding box of each symbol. S132, search for maximal sub-formulas with the same structure in the online handwritten mathematical formula library, and if found, adjust the position of the outer square of the symbol according to the handwritten sample. S133, replace each symbol with one of the corresponding handwritten symbols in the online handwritten mathematical formula library, and then perform a slight random affine transformation; S134, randomly rotates and scales the synthesized handwriting.

6. The syntax-guided end-to-end trainable online handwritten mathematical formula recognition method according to claim 1, characterized in that, The database used to train the model includes: online handwritten mathematical formula samples, offline handwritten mathematical formula samples, printed mathematical formula samples, mathematical formula corpus, and randomly generated mathematical formulas.

7. The syntax-guided end-to-end trainable online handwritten mathematical formula recognition method according to claim 6, characterized in that, Training is based on data sources in the following forms: A sample library of online handwritten mathematical formulas, where each online handwritten mathematical formula is saved in InkML format, and also includes annotations saved in MathML format; A library of offline handwritten and printed mathematical formula samples, each of which includes an image saved in PNG format and annotations saved in MathML format; A corpus of mathematical formulas, where each formula is saved in MathML format.

8. The syntax-guided end-to-end trainable online handwritten mathematical formula recognition method according to claim 1, characterized in that, During the decoding process, ensure that the identified mathematical formulas have string representations that conform to the given syntax; combine the predictive analysis method used to parse LL(1) syntax with beam search to complete the mathematical formula decoding process.

Images