A mathematical formula recognition method and system for handwritten answers to a topic prompt

Abstract

This invention discloses a method and system for recognizing handwritten mathematical formulas in response to prompts. The method includes the following steps: obtaining an image of a handwritten mathematical formula; extracting visual features from the image; identifying the prompt corresponding to the image; extracting first prompt text features from the prompt; extracting second prompt text features related to the visual features from the first prompt text features using a scaling dot product attention mechanism; fusing the visual features and the second prompt text features using a gating mechanism to obtain fused features; and decoding the fused features using a GRU decoder with an attention mechanism to output a formula label sequence. This invention can improve the accuracy of recognizing handwritten mathematical formulas in response to prompts.

Description

Technical Field

[0001] This invention belongs to the field of pattern recognition, and more specifically, relates to a method and system for recognizing handwritten mathematical formulas in response to prompts. Background Technology

[0002] Handwritten mathematical formulas are characterized by diverse writing styles, numerous symbols, and complex two-dimensional structures, making handwritten mathematical formula recognition tasks extremely challenging. Because students, especially primary and secondary school students, have not yet fully developed their handwriting skills, their writing often exhibits characteristics such as sloppiness and irregularities, making handwritten mathematical formula recognition in educational scenarios even more difficult than in general scenarios.

[0003] Research on mathematical formula recognition originated in the 1860s. In its early stages, researchers typically followed a process of symbol segmentation, symbol recognition, and structural analysis. This process first generated a set of symbolic hypotheses, then used parsing techniques to generate valid mathematical expressions. These methods not only suffered from error accumulation but also required the manual design of numerous grammatical production rules. In recent years, with the rapid development of deep learning technology, encoder-decoder frameworks have been extensively explored in the field of handwritten mathematical formula recognition. Researchers treat handwritten mathematical formula recognition as an image-to-sequence task, directly transcribing handwritten mathematical formula images into corresponding LaTeX labeled sequences. Encoder-decoder-based methods, benefiting from end-to-end trainability, significantly outperform traditional methods.

[0004] Solutions proposed for mathematical formula recognition can be broadly categorized into three types: sequential solutions, ensemble solutions, and solutions based on end-to-end trainable neural networks. In sequential solutions, symbol segmentation, symbol recognition, and structural analysis are assumed to be independent tasks. However, this assumption contradicts the fact that these three tasks are interdependent. To resolve this contradiction, researchers have turned to ensemble solutions. These solutions utilize contextual information (here, grammatical knowledge) to guide symbol segmentation and recognition, thereby avoiding the generation of mathematical formulas that do not conform to grammatical rules. However, ensemble solutions require manually designing a large number of grammatical production rules for different datasets; furthermore, the time complexity of syntax parsing algorithms is relatively high.

[0005] Currently, the mainstream methods for handwritten mathematical formula recognition belong to the third category. These methods are mostly based on encoder-decoder frameworks, treating mathematical formula recognition as an image-to-sequence task, and have achieved excellent results. In general, these deep learning-based methods either focus on optimizing the encoder or decoder or attempt to introduce additional information such as counting information or grammatical rules to improve recognition accuracy. However, none of these methods can solve the recognition ambiguity problem caused by illegible or non-standard handwriting in educational scenarios. Summary of the Invention

[0006] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a method and system for recognizing handwritten mathematical formulas in question prompts, which can improve the accuracy of recognizing handwritten mathematical formulas in question prompts.

[0007] To achieve the above objectives, according to one aspect of the present invention, a method for recognizing handwritten mathematical formulas in problem prompts is provided, comprising the steps of:

[0008] Obtain a handwritten mathematical formula image, extract visual features from the handwritten mathematical formula image, identify the question corresponding to the handwritten mathematical formula image, and extract the first question text features from the question;

[0009] A scaling dot product attention mechanism is used to extract second question text features related to the visual features from the first question text features. The visual features and the second question text features are then fused to obtain fused features.

[0010] A GRU decoder with an attention mechanism is used to decode the fused features and output a formula label sequence.

[0011] Furthermore, the visual features are extracted from the handwritten mathematical formula image using the DenseNet network.

[0012] Furthermore, the extraction of the first question text features includes the following steps:

[0013] Identify the mathematical symbols and formulas contained in the problem, and record the identification result as Q. m Using the embedding matrix multiplied by Q m The text features of the first question are obtained.

[0014] Furthermore, the step of extracting the second question text features related to the visual features from the first question text features using the scaled dot product attention mechanism includes the following steps: using the visual features as the query vector and the first question text features as the key and attribute values, calculating the second question text features related to the visual features, using the following formula:

[0015] Q = W q (F)

[0016] K = W k (L)

[0017] V = W v (L)

[0018] Z = Attention(Q, K, V)

[0019] Among them, W q W k W v All are projection functions, F is the visual feature, L is the first question text feature, Attention(·) represents the attention function, and Z is the second question text feature.

[0020] Furthermore, a gating unit is used to control the feature fusion of the visual features and the second question text features, and the calculation formula is as follows:

[0021] F f =g(Z)⊙Z+F

[0022] Where ⊙ represents element-wise multiplication, g(·) represents the gate unit, and F f Z represents the fusion feature, F represents the second title text feature, and F represents the visual feature.

[0023] Furthermore, the gating unit includes a two-layer perceptron, the first layer being a linear layer and a ReLU nonlinear activation function, and the second layer being a linear layer and a Tanh activation function.

[0024] According to another aspect of the present invention, a handwritten mathematical formula recognition system with prompts is provided, comprising:

[0025] The feature extraction module is used to acquire handwritten mathematical formula images, extract visual features from the handwritten mathematical formula images, identify the questions corresponding to the handwritten mathematical formula images, and extract first question text features from the questions.

[0026] The feature fusion module is used to extract second question text features related to the visual features from the first question text features using a scaled dot product attention mechanism, and to fuse the visual features with the second question text features to obtain fused features.

[0027] The decoding module is used to decode the fused features using a GRU decoder with an attention mechanism and output a formula label sequence.

[0028] Furthermore, the feature extraction module includes a question text feature extraction module, used to identify mathematical symbols and formulas contained in the question, and denoting the identification result as Q. m Using the embedding matrix multiplied by Q m The text features of the first question are obtained.

[0029] Furthermore, the step of extracting the second question text features related to the visual features from the first question text features using the scaled dot product attention mechanism includes the following steps: using the visual features as the query vector and the first question text features as the key and attribute values, calculating the second question text features related to the visual features, using the following formula:

[0030] Q = W q (F)

[0031] K = W k (L)

[0032] V = W v (L)

[0033] Z = Attention(Q, K, V)

[0034] Among them, W q W k W v Let F be the projection function, F be the visual feature as the query vector, L be the first question text feature, Attention(·) represent the attention function, and Z be the second question text feature.

[0035] Furthermore, the feature fusion module includes a gating unit, which controls the feature fusion of the visual features and the second question text features, calculated using the following formula:

[0036] F f =g(Z)⊙Z+F

[0037] Where ⊙ represents element-wise multiplication, g(·) represents gating mechanism, and F f Z represents the fusion feature, F represents the second title text feature, and F represents the visual feature.

[0038] Overall, compared with the prior art, the above-described technical solutions conceived in this invention propose a method for recognizing handwritten mathematical formulas based on the strong correlation between questions and answers. By introducing question information in the encoding stage, this method solves the problem that existing encoder-decoder-based methods cannot effectively utilize question text information, thereby improving the accuracy of handwritten mathematical formula recognition and achieving direct and effective transcription from images to LaTeX. Attached Figure Description

[0039] Figure 1 This is a flowchart of the handwritten mathematical formula recognition method according to an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram illustrating the principle of the handwritten mathematical formula recognition method and system according to an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0042] In the description of the embodiments of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. The terms "comprising" and "having," and any variations thereof, in the embodiments of this invention are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices.

[0043] The naming or numbering of steps in the embodiments of the present invention does not mean that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved.

[0044] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0045] This invention provides a method and system for recognizing handwritten mathematical formulas in problem prompts, which will be described below.

[0046] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a method for recognizing handwritten mathematical formulas in response to prompts, comprising the following steps:

[0047] S101, Feature extraction, namely, acquiring a handwritten mathematical formula image, extracting visual features from the handwritten mathematical formula image, identifying the question corresponding to the handwritten mathematical formula image, and extracting the first question text features from the question. The following is a detailed explanation.

[0048] There is a correspondence between the questions and the handwritten mathematical formula images. Unlike existing technologies, in this embodiment of the invention, in addition to directly extracting visual features from the handwritten mathematical formula images, the invention also makes full use of the questions corresponding to the handwritten mathematical formula images to extract the first question text features from the questions.

[0049] In one embodiment, the specific method for identifying the question corresponding to the handwritten mathematical formula image is as follows: acquire the image of the handwritten test paper, use object detection technology (such as YOLO) to extract the question area and the answer area from the input image, and use the spatial position relationship between the areas to associate the question with the answer; the question is printed, and OCR technology is used to identify the question text.

[0050] There are various methods for extracting visual features from the handwritten mathematical formula image and extracting the first question text features from the question, and the embodiments of the present invention do not limit them.

[0051] In one embodiment, a given handwritten mathematical formula image and its corresponding question text are obtained. Preferably, two different models are used to extract visual features and question text features respectively.

[0052] (1) Visual Features. This embodiment of the invention uses DenseNet to extract visual features from images of handwritten mathematical formulas. Its output is a three-dimensional feature map. Where H, W, and C represent height, width, and number of channels, respectively.

[0053] (2) Problem Text Features. In this embodiment of the invention, the correlation between the problem Q and the handwritten mathematical formula is simplified to the mathematical symbols or mathematical expressions Q contained in the problem. m Relevance to handwritten mathematical formulas. These symbols or expressions are further represented as LaTeX sequences for processing. Therefore, Q m In addition to the visible mathematical symbols, there are also some auxiliary symbols, such as {,}, ∧, - Etc. Embedding matrices can be used. Embed Qm into a high-dimensional vector This serves as the textual feature of the first question.

[0054] L=W e Q m (1)

[0056] Where E represents the size of the token embedding, S represents the number of all symbol classes, and Tq is the simplified problem text Q. m The length of the embedding matrix W. e It can be determined through training.

[0057] In another embodiment, in addition to considering the mathematical symbols or mathematical expressions Q contained in the problem... m In addition to handwritten mathematical formulas, the natural language text contained in the problem will also be considered. Specifically, the entire problem text Q (including natural language text and mathematical language text) is input into the BERT pre-trained model to obtain the text features L of the problem.

[0058] L = BERT(Q)

[0059] S102, Feature fusion, that is, using a scaling dot product attention mechanism to extract second question text features related to the visual features from the first question text features, and fusing the visual features with the second question text features to obtain fused features.

[0060] To model the interaction between the visual information of handwritten mathematical formulas and the text information of the problem, this invention proposes a feature fusion module based on a scaled dot product attention mechanism. Unlike traditional multimodal fusion tasks, the problem text and the mathematical formula image do not describe the same thing. In most cases, only part (not all) of the information in the problem text is useful for the recognition task. Therefore, this invention employs a scaled dot product attention mechanism to calculate Q based on visual features. m The importance of different markers. Specifically, using visual features F as the query vector, and the first question text feature L as the key and attribute value, the question text feature Z related to the visual features can be calculated by the following formula:

[0061] Q = W q (F) (2)

[0063] K = W k (L) (3)

[0065] V = W v (L) (4)

[0067]

[0068] Among them W q W k W v The projection functions, W, can all be determined through training. k and W v Using 1×1 convolution, while W q We employ 1×1 convolution and instance normalization, where Attention(.) represents the attention function, Z is the second question text feature, softmax(.) represents the softmax function, and d kLet K represent the dimension, and T represent the transpose.

[0069] Preferably, the question text can only serve as a prompt for the recognition task when needed. If the visual features are clear enough to support the recognition process, no further prompt is required. Therefore, to prevent text features from having an unnecessary negative impact on visual features, this patent employs a gating unit to control the fusion of visual and text features, as shown in the following formula:

[0070] F f =g(Z)⊙Z+F (6)

[0072] Where ⊙ denotes element-wise multiplication, and g(·) denotes a gating unit. Furthermore, this gating unit employs a two-layer perceptron: the first layer is a linear layer with a ReLU nonlinear activation function, and the second layer is a linear layer with a Tanh activation function. F f This indicates the characteristics after fusion.

[0073] S103, using a GRU decoder with an attention mechanism, decodes the fused features and outputs a formula label (LATEX) sequence.

[0074] Specifically, the GRU decoder is used to progressively output the decoded symbols corresponding to the handwritten mathematical formula image. The GRU decoder includes a feature attention module, a gated recurrent unit, and a prediction module. The feature attention module is used to calculate the attention weight of the current step based on the hidden state of the previous step and the fused features, and to calculate the context vector of the current step based on the attention weight of the current step and the fused features. The gated recurrent unit is used to calculate the hidden state of the current step based on the decoded symbol output from the previous step, the context vector of the current step, and the hidden state of the previous step. The prediction module is used to calculate and output the decoded symbol of the current step based on the context vector of the current step, the hidden state of the current step, and the decoded symbol output from the previous step.

[0075] This invention employs a GRU decoder with an attention mechanism to decode the fused features, progressively generating a LaTeX sequence. The decoder output is a LaTeX tag sequence corresponding to a handwritten mathematical formula. This sequence is not output all at once, but generated progressively. If the sequence contains T characters, the entire decoding process consists of T steps, with the current step denoted as step t and the previous step as step t-1. Decoding at step t depends on the information from step t-1.

[0076] At step t, the GRU decoder utilizes the hidden state h from the previous step. t-1 Fusion feature F f And calculate the current attention weight α with the coverage attention A t and context vector ct:

[0077]

[0078]

[0079]

[0080] Where v a W h W a For trainable weights, U f For a 1×1 convolution operation, A is the sum of all attention weights from the past t-1 steps, i.e., the sum of attention weights from step 1, step 2, ... up to step t-1. t，ij For e t The (i, j)th element, e t，pq For e t The (p, q)th element in the diagram, where tanh(·) represents the activation function, and α t，ij For a t The (i, j)th element, α t，i It is α t The i-th element, which is also α t The i-th element after transformation into a one-dimensional vector, F fi For F f The i-th vector in the sequence, where T represents the transpose. The previous hidden state h... t-1 It is the output calculated by the gated loop unit when decoding the (t-1)th character.

[0081] Then, the symbols obtained from decoding in step t-1 are embedded into E(y). t-1 ) and context vector c t The input is fed into the GRU (Gated Loop Unit) to obtain the current hidden state h. t , where y t-1 The decoded tag output in the previous step is embedded in E(y) t-1 ) is to use y t-1 Multiply by a trainable embedding matrix:

[0082] h t =GRU(E(y) t-1 ), ct, h t-1 ) (10)

[0084] Finally, predict the symbol probability p(y) t )for:

[0085]

[0086] Where w o W e W h′ Wc These are trainable weights. p(y) t The output symbol y at step t is... t The corresponding probabilities. At each step, the symbol with the highest probability is selected as the decoding symbol for that step. The decoded symbols output by all steps are the LATEX sequence of the corresponding symbols.

[0087] According to a second aspect of the present invention, a handwritten mathematical formula recognition system with prompts is provided, comprising: a feature extraction module, a feature fusion module, and a decoding module.

[0088] The feature extraction module is used to acquire handwritten mathematical formula images, extract visual features from the handwritten mathematical formula images, identify the questions corresponding to the handwritten mathematical formula images, and extract first question text features from the questions.

[0089] The feature fusion module is used to extract second question text features related to the visual features from the first question text features using a scaled dot product attention mechanism, and to fuse the visual features with the second question text features to obtain fused features.

[0090] The decoding module is used to decode the fused features using a GRU decoder with an attention mechanism and output a formula label sequence.

[0091] The entire system (model) is trainable end-to-end and uses the cross-entropy classification loss function.

[0092] The specific implementation and technical effects of the system are the same as those described above, and will not be repeated here.

[0093] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for recognizing handwritten mathematical formulas in problem prompts, characterized in that, Including the following steps: Obtain a handwritten mathematical formula image, extract visual features from the handwritten mathematical formula image, identify the question corresponding to the handwritten mathematical formula image, and extract the first question text features from the question; A scaling dot product attention mechanism is used to extract second question text features related to the visual features from the first question text features. The visual features and the second question text features are then fused to obtain fused features. A GRU decoder with an attention mechanism is used to decode the fused features and output a formula label sequence; The step of extracting second question text features related to the visual features from the first question text features using the scaled dot product attention mechanism includes the following steps: using the visual features as the query vector and the first question text features as the key and attribute values, calculating the second question text features related to the visual features, using the following formula: in, Both are projection functions. For the aforementioned visual features, The text features of the first question. Represents the attention function. This refers to the text features of the second question.

2. The method for recognizing handwritten mathematical formulas in question prompts as described in claim 1, characterized in that, The visual features are extracted from the handwritten mathematical formula image using the DenseNet network.

3. The method for recognizing handwritten mathematical formulas in question prompts as described in claim 1, characterized in that, The steps for extracting the text features of the first question include: Identify the mathematical symbols and formulas contained in the problem, and record the identification results as follows: Using embedding matrix multiplication The text features of the first question are obtained.

4. The method for recognizing handwritten mathematical formulas in question prompts as described in claim 1, characterized in that, A gating unit is used to control the feature fusion of the visual features and the second question text features. The calculation formula is as follows: Where ⊙ denotes element-wise multiplication. This refers to the gate control unit. This indicates the fusion feature. This indicates the textual features of the second question.

5. The method for recognizing handwritten mathematical formulas in question prompts as described in claim 4, characterized in that, The gated unit includes two layers of perceptrons: the first layer is a linear layer with a ReLU nonlinear activation function, and the second layer is a linear layer with a Tanh activation function.

6. A handwritten mathematical formula recognition system with prompts, characterized in that, include: The feature extraction module is used to acquire handwritten mathematical formula images, extract visual features from the handwritten mathematical formula images, identify the questions corresponding to the handwritten mathematical formula images, and extract first question text features from the questions. The feature fusion module is used to extract second question text features related to the visual features from the first question text features using a scaled dot product attention mechanism, and to fuse the visual features with the second question text features to obtain fused features. The decoding module is used to decode the fused features using a GRU decoder with an attention mechanism and output a formula label sequence; The step of extracting second question text features related to the visual features from the first question text features using the scaled dot product attention mechanism includes the following steps: using the visual features as the query vector and the first question text features as the key and attribute values, calculating the second question text features related to the visual features, using the following formula: in, Both are projection functions. For the aforementioned visual features, The text features of the first question. Represents the attention function. This refers to the text features of the second question.

7. The handwritten mathematical formula recognition system for problem prompts as described in claim 6, characterized in that, The feature extraction module includes a question text feature extraction module, used to identify mathematical symbols and formulas contained in the question, and record the identification results as follows: Using embedding matrix multiplication The text features of the first question are obtained.

8. The handwritten mathematical formula recognition system for problem prompts as described in claim 6, characterized in that, The feature fusion module includes a gating unit, which controls the feature fusion of the visual features and the second question text features. The calculation formula is as follows: Where ⊙ denotes element-wise multiplication. Indicates gating mechanism, This indicates the fusion feature. This indicates the textual features of the second question.

Images