Graph structure guidance-based stylized handwritten formula generation method

Abstract

The invention relates to the technical field of computer vision and pattern recognition, in particular to a stylized handwritten formula generation method based on graph structure guidance, which comprises the following steps: step 1, based on a target symbol dictionary, automatically generating mathematical formulas in batches by utilizing a python script and constructing a corresponding latex sequence; step 2, converting the latex sequence into a symbol graph; step 3, using a layout predictor based on a graph convolutional network to predict a rough layout of each symbol according to the symbol graph position information, and generating a bounding box; step 4, mapping the bounding box to a pixel-level soft mask; and 5, generating a handwritten mathematical expression image according to the soft mask by using an image decoder. According to the handwritten formula generation method, reliable data of a large batch of handwritten formulas can be quickly, efficiently and automatically generated, the problem that manual annotation and drawing acquisition are depended currently is effectively solved, and the data cost is remarkably reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and pattern recognition, and in particular to a method for generating stylized handwritten formulas based on graph structure guidance. Background Art

[0002] HME (Handwritten mathematical expressions) has a wide range of applications in education and scientific research. However, due to the structural complexity and writing diversity of handwritten mathematical expressions, automatic recognition of handwritten mathematical expressions (HMER) remains a challenging task. Existing methods mainly rely on a large amount of real data for training, while traditional data collection mainly relies on a large amount of manual annotation and drawing, which is costly and difficult to obtain on a large scale. Therefore, developing a method that can generate high-quality handwritten mathematical expression images is of great significance to improving the performance of HMER systems. Summary of the invention

[0003] In order to solve the above technical problems existing in the prior art, the present invention proposes a stylized handwritten formula generation method based on graph structure guidance, which can realize automatic generation of handwritten mathematical formulas and provide strong support for the training of handwritten formula recognition model. The specific technical solution is as follows: A method for generating stylized handwritten formulas based on graph structure guidance includes the following steps: Step 1: Generate mathematical formulas and construct corresponding latex sequences based on the target symbol dictionary; Step 2: Convert the latex sequence into a symbolic graph; Step 3: Use the layout predictor based on graph convolutional network to predict the rough layout of each symbol according to the symbol graph position information and generate the bounding box; Step 4: Map the bounding box to a pixel-level soft mask; Step 5: Use the image decoder to generate a handwritten mathematical expression image based on the soft mask.

[0004] Furthermore, in step 1, a python script is used to batch generate random mixed operation mathematical formulas, and then the formulas are converted into LaTeX format to obtain corresponding latex sequences.

[0005] Furthermore, in step 2, each node in the symbol graph represents a mathematical symbol, and each edge represents a positional relationship between mathematical symbols.

[0006] Furthermore, in step 3, the layout predictor is trained in an adversarial learning manner to obtain a four-dimensional rectangular box [x0, y0, x1, y1] in the [0, 1] coordinate space, where (x0, y0) and (x1, y1) respectively represent the coordinates of the upper left corner and lower right corner of the boundary of the bounding box, accurately predicting the size of each symbol.

[0007] Furthermore, in step 4, the soft mask first uses two convolutional layers to predict the low-resolution mask, and then uses two upsampling convolutional layers to upsample the mask in sequence.

[0008] Furthermore, in step 5, the image decoder adopts a cascade refinement network, and introduces an image discriminator and a symbol discriminator to respectively evaluate the overall quality of the generated image and the accuracy of local symbols.

[0009] Furthermore, a style generation module is introduced to generate handwritten formula images in different user styles.

[0010] The handwritten formula generation method of the present invention can quickly, efficiently and automatically generate reliable data of large quantities of handwritten formulas, so as to utilize the handwritten formulas to train a deep learning model used to recognize handwritten formulas, effectively solving the current problem of relying on manual annotation and drawing collection, and significantly reducing data costs; especially compared with natural image generation, handwritten formula generation is more dependent on structured layout, and the clear layout in the present invention plays a vital role in generating high-quality handwritten formulas, and accurately predicts the size of each symbol and the positional relationship between them. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a flowchart of a method for generating stylized handwritten formulas based on graph structure guidance according to this embodiment. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical scheme and technical effect of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0013] like Figure 1 As shown, a method for generating a stylized handwritten formula based on graph structure guidance in this embodiment includes the following steps: Step 1, latex sequence generation: According to the target symbol dictionary (1, +, =, 2), that is, the symbol content involved in the formula, use the python script to batch generate mathematical formulas (1+1=2) and construct the corresponding latex sequence (1+1=2).

[0014] Two functions are designed using Python scripts to generate latex sequences. The function first generates an operand list containing random integers (1 1 2), then selects a random operator, (+) is selected as the operator in this example, and constructs an expression string (1+1=2). At the same time, an additional check is set to avoid division by 0 and prevent the same multiplication or division operator from appearing consecutively. After that, the generated expression is converted to LaTeX format, which replaces the multiplication operator * with \times and the division operator / with \div. Finally, a LaTeX string wrapped with a $ symbol is returned for direct use in an environment that supports LaTeX rendering.

[0015] Step 2: Symbolic graph construction: Input a latex sequence and parse it into a symbolic graph, that is, build a graph structure based on the latex sequence to describe the relative positional relationship of each symbol in a mathematical formula. Each node in the symbolic graph represents a mathematical symbol, and each edge represents the positional relationship between mathematical symbols. Use a learnable embedding layer to map nodes and edges into high-dimensional feature vectors. Given a set of mathematical symbol categories C and a set of relationship categories R, a symbolic graph (S, E) can be constructed based on C and R. Each node in S={S1, ···, Sn} is a mathematical symbol, where node si∈C, n is the number of mathematical symbols; and each directed edge (i, j) in E={eij} represents the positional relationship of node j wrt (With Respect To, about) node i, where eij=(si, r, sj), r∈R. For example, given an expression x 2 , which can be represented by a tuple (x, sup, 2), where sup is an abbreviation for superscript.

[0016] Step 3: Layout prediction: A layout predictor based on a graph convolutional network is used to predict the rough layout of each symbol based on the symbol graph position information and generate a bounding box to guide the position layout when generating formulas. The layout predictor is trained in an adversarial learning manner to obtain a four-dimensional rectangular box [x0, y0, x1, y1] in the [0, 1] coordinate space to generate a predicted bounding box for the subsequent mask. (x0, y0) and (x1, y1) represent the coordinates of the upper left corner and lower right corner of the bounding box boundary, respectively.

[0017] Step 4: Layout refinement: A sequential transformation module is designed to map the bounding box to a soft mask at the pixel level, that is, to perform mask inference to eliminate the blurred areas in the layout. First, two convolutional layers are used to predict a low-resolution mask of dimension w × w, and then two upsampling convolutional layers are used to upsample the mask in sequence, with an upsampling factor of 2. These masks are used in the image decoder to generate images of the same size, which are used to guide the formula to generate the layout positions in the figure, that is, five in 1 + 1 = 2.

[0018] Step 5: Image generation: A cascaded refinement network is used as an image decoder to generate handwritten mathematical expression images based on a soft mask. In order to improve the realism of the generated images, an image discriminator and a symbol discriminator are introduced to evaluate the overall quality of the generated images and the accuracy of local symbols, respectively. Gaussian noise is added to the feature map to improve the diversity of image generation, and a style generation module is introduced to guide its stylized generation. Finally, handwritten formula images with different user styles are generated.

[0019] The above is only a preferred implementation case of the present invention and does not limit the present invention in any form. Although the implementation process of the present invention is described in detail above, for those familiar with the art, they can still modify the technical solutions recorded in the above examples, or replace some of the technical features therein with equivalents. All modifications, equivalent replacements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating stylized handwritten formulas based on graph structure guidance, characterized in that: The following steps are involved: Step 1: Generate mathematical formulas and construct corresponding latex sequences based on the target symbol dictionary; Step 2: Convert the latex sequence into a symbolic graph; Step 3: Use the layout predictor based on graph convolutional network to predict the rough layout of each symbol according to the symbol graph position information and generate the bounding box; Step 4: Map the bounding box to a pixel-level soft mask; Step 5: Use the image decoder to generate a handwritten mathematical expression image based on the soft mask.

2. The method for generating stylized handwritten formulas according to claim 1, characterized in that: In the step 1, a python script is used to batch generate random mixed operation mathematical formulas, and then the formulas are converted into LaTeX format to obtain corresponding latex sequences.

3. The method for generating stylized handwritten formulas according to claim 1, characterized in that: In step 2, each node in the symbol graph represents a mathematical symbol, and each edge represents the positional relationship between mathematical symbols.

4. The method for generating stylized handwritten formulas according to claim 1, characterized in that: In step 3, the layout predictor is trained in an adversarial learning manner to obtain a four-dimensional rectangular box [x0, y0, x1, y1] in the [0, 1] coordinate space, where (x0, y0) and (x1, y1) represent the coordinates of the upper left corner and the lower right corner of the boundary of the bounding box, respectively.

5. The method for generating stylized handwritten formulas according to claim 1, characterized in that: In step 4, the soft mask first uses two convolutional layers to predict a low-resolution mask, and then uses two upsampling convolutional layers to upsample the mask in sequence.

6. The method for generating stylized handwritten formulas according to claim 1, characterized in that: In step 5, the image decoder adopts a cascade refinement network, and introduces an image discriminator and a symbol discriminator to respectively evaluate the overall quality of the generated image and the accuracy of the local symbols.

7. The method for generating stylized handwritten formulas according to claim 6, characterized in that: A style generation module is also introduced to generate handwritten formula images in different user styles.

Images