Data enhancement method for mathematical formula identification model

By performing data preprocessing and character replacement on the CROHME data set, combined with fractional structure interchange and deformation, the problem of overfitting the existing mathematical formula recognition model is solved, and the generalization ability and recognition accuracy of the model are improved.

CN119049064APending Publication Date: 2024-11-29DUT ARTIFICIAL INTELLIGENCE INST DALIAN +1

Patent Information

Application Number
CN202411217447.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-02
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The existing mathematical formula recognition model is easy to overfit and difficult to generalize to new samples due to limited training data. The traditional data augmentation method fails to fully cover the semantic diversity of mathematical formulas, resulting in a decline in generalization performance.

Method used

By obtaining the online formula data in the CROHME dataset, data preprocessing includes noise reduction and normalization, obtaining the categories and positions of each character, character replacement and content exchange between the upper and lower parts of the fractional structure, and scaling and rotating the formula. Update the LaTeX sequence to increase the semantic diversity of data samples and the generalization ability of the model.

Benefits of technology

It effectively removes data noise, increases the semantic diversity of data samples, improves the generalization ability of the model, reduces overfitting, and improves the accuracy of identification of complex mathematical formulas.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119049064A_ABST
    Figure CN119049064A_ABST
Patent Text Reader

Abstract

The invention relates to a data enhancement method for a mathematical formula recognition model, and the method comprises the following steps: obtaining online formula data in a CROHME data set, and obtaining the type and position of each character in the online formula data; performing data preprocessing on samples in the online formula data; and judging each character in the online formula data, performing character replacement, performing scaling and rotation transformation on the whole formula, and updating the LaTeX sequence corresponding to the current formula. The method is a data enhancement method based on a character annotation data set, the characteristic that a CROHME online data set has character positions and category annotations is fully utilized, through noise reduction processing and standardization, data noise is removed, the influence of error information on a model is reduced, model learning and feature extraction are easier, and the method has the advantages of being high in practicability and easy to popularize. According to the method, local characters and global formulas are deformed, different styles of handwritten formulas are simulated, semantic information of the formulas is changed, the diversity of data sample semantics is expanded, and the generalization ability of the model is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of formula recognition, and in particular to a data enhancement method for a mathematical formula recognition model. Background Art

[0002] Mathematical formula recognition technology is a key technology in the field of electronic transcription of paper documents. Its significance and application value cannot be ignored. Since mathematical formulas usually contain complex structures and symbols, such as superscripts and subscripts, fraction lines, square roots, and combinations of various mathematical symbols, mathematical formulas are difficult to enter directly into electronic devices. Especially when it comes to formulas in handwritten or printed documents, traditional text input methods and optical character recognition technology are often difficult to accurately recognize and process.

[0003] Mathematical formula recognition research is not only of great value in the academic and educational fields, but also provides valuable experience and technical accumulation for the advancement of optical character recognition technology in other fields. The challenges faced by handwriting recognition technology include handwriting diversity, individual differences in writing styles, etc. The solution to these problems is of guiding significance for improving the accuracy and robustness of optical character recognition technology, and also promotes the technological progress of the entire optical character recognition field.

[0004] Traditional mathematical formula recognition usually includes three main parts: character segmentation, character recognition, and structural reorganization. The LaTeX sequence of mathematical formulas is usually much longer than the text in the image description task. This difference mainly stems from the complexity and flexibility of mathematical formulas. Compared with objects described in natural language, mathematical formulas usually contain more symbols, operators, and structures. Therefore, in order to accurately represent mathematical formulas, a longer sequence length is required to capture its rich information and structure, which is a significant feature compared with the image description task. Conventional encoder-decoder frameworks usually have difficulty dealing with long sequence problems, and attention mechanisms can deal with this situation well, because the attention mechanism can make the model pay attention to what should be paid attention to when decoding. Although the attention mechanism can alleviate the problems caused by long sequences to the encoder-decoder model to a certain extent, the conventional attention mechanism may have problems of over-attention or lack of attention, that is, the model repeatedly pays attention to a certain feature in the decoding stage, causing the model to make incorrect decoding outputs.

[0005] Mathematical formula recognition models based on deep learning usually rely on datasets. IM2LATEX-100K is a well-known dataset for printed mathematical formula recognition, which provides a large number of training samples, thus facilitating the training of complex deep learning models. The CROHME dataset is used for handwritten mathematical formula recognition, and its sample size is relatively small, with about 10K training samples and high diversity. Due to the limitation of training data, mathematical formula models may over-learn specific patterns and noise in these limited samples and fail to effectively generalize to new, unseen samples. Therefore, we need data augmentation technology to enhance and denoise some existing datasets, and increase the quantity and diversity.

[0006] Data augmentation is a commonly used method to combat overfitting by artificially increasing the diversity of training data. In offline handwritten mathematical formula image recognition, people have adopted a variety of data augmentation methods, such as affine transformation, perspective transformation, and random elastic deformation, to generate new training samples. These methods can increase the diversity of data to a certain extent and improve the generalization ability of the model. However, although data augmentation helps the model better learn the representation of mathematical formulas under changes and perturbations, current methods still have limitations. In particular, if data augmentation techniques mainly increase the visual diversity of samples without fully covering the semantic diversity of mathematical formulas, the model may over-learn the specific semantic content in the training samples, resulting in a decrease in generalization performance on complex or unseen mathematical formulas. This phenomenon is called overfitting of formula semantic information. Summary of the invention

[0007] The technical problem to be solved by the present invention is to overcome the deficiencies in the prior art and provide a data enhancement method for a mathematical formula recognition model.

[0008] The present invention is achieved through the following technical solutions:

[0009] A data enhancement method for a mathematical formula recognition model comprises the following steps:

[0010] S1. Obtain online formula data in the CROHME dataset, and obtain the category and position of each character in the online formula data;

[0011] S2. Performing data preprocessing on samples in the online formula data;

[0012] S3. Determine each character in the online formula data and replace the characters;

[0013] S4. Scale and rotate the entire formula after character replacement, and update the LaTeX sequence corresponding to the current formula.

[0014] According to the above technical solution, preferably, in step S2, the data preprocessing includes data noise reduction and data normalization.

[0015] According to the above technical solution, preferably, in step S2, the data noise reduction includes: deleting closely spaced coordinate points and points densely distributed along a straight line.

[0016] According to the above technical solution, preferably, in step S2, the data normalization includes: calculating the projection center of the coordinate point on the horizontal and vertical axes; calculating the standard deviation of the projection center; and normalizing the coordinate point based on the projection center and the standard deviation.

[0017] According to the above technical solution, preferably, in step S2, the projection center of the coordinate point on the horizontal and vertical axes is calculated.

[0018]

[0019] Among them: Ω represents the set of all line segments in the mathematical formula,

[0020] represents the length of the line segment L formed by (x1, y1) and (x2, y2),

[0021] p"(L) and p$(L) represent the projection lengths of line segment L on the x-axis and y-axis, respectively, namely:

[0022]

[0023] According to the above technical solution, preferably, in step S2, the standard deviation of the projection center is calculated.

[0024] d x (L) = ∫ L (x-μ x ) 2 dL,

[0025] Get the standard deviation on the x-axis,

[0026]

[0027] According to the above technical solution, preferably, in step S2, the coordinate points are normalized based on the projection center and the standard deviation, wherein:

[0028]

[0029] According to the above technical solution, preferably, in step S3, if it is a fraction separator, it is put into the fraction array to swap the contents of the upper and lower parts; if it is a character in the replaceable character group, the character is replaced.

[0030] According to the above technical solution, preferably, in step S3, the replaceable character groups are classified, the character replacement is performed in the same character group, and then all the characters are randomly combined and changed from multiple types of shearing, scaling, rotation and perspective transformation.

[0031] The beneficial effects of the present invention are:

[0032] The present invention is a data enhancement method based on a character annotation dataset, which makes full use of the characteristics of the CROHME online dataset with character position and category annotations. First, the CROHME data is subjected to noise reduction and normalization to remove the noise of the data, reduce the influence of the model on the error information, make it easier for the model to learn and extract features, and also facilitate the subsequent character replacement and the exchange of the upper and lower contents of the fraction structure;

[0033] The same type of characters in existing formulas are randomly replaced, and the upper and lower contents are swapped when encountering a fractional structure, which changes the semantic information of the formula and expands the semantic diversity of the data samples; local characters and global formulas are deformed to simulate the different styles of handwritten formulas, which helps to improve the generalization ability of the model; the LaTeX tags of the formula images are normalized to remove useless information and avoid the situation where the same feature corresponds to multiple outputs. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic flow chart of the method of the present invention.

[0035] Figure 2 It is a classification diagram of replaceable character groups in the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and the best embodiment. Based on the embodiments in the invention, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the invention.

[0037] The main challenges in building high-quality datasets lie in the process of data collection, manual annotation, and data validation. First, data collection requires a lot of time and resources, especially when the data sources are scattered or difficult to obtain. Second, manual annotation is a time-consuming and labor-intensive process, especially for complex data that requires expertise to accurately annotate, such as mathematical formulas. Finally, data validation ensures the accuracy and consistency of the dataset, but this is also a resource-intensive step.

[0038] In order to effectively address these challenges, this application adopts the method of determining sample labels before collecting them. This strategy allows us to define in advance the sample types and features that we want to include in the dataset, so that we can collect and label data in a more targeted manner. This method not only reduces the workload of subsequent labeling, but also helps to better control the sample distribution and ensure that the dataset can reflect the distribution of sample characteristics required for scientific research. In this way, a more representative and scientifically valuable dataset can be constructed, providing a solid foundation for research.

[0039] In the scenario of handwritten mathematical formula recognition, the selection of LaTeX tags is modified based on the tags of the CROHME training set samples, which cover handwritten samples of volunteers of different age groups and educational backgrounds, ensuring data diversity and wide applicability. This patent has obtained a total of 10,200 formula images. The image generation process includes scanning the paper through scanning software and subsequent cropping. These steps ensure the quality of the image and provide high-quality input for model training and evaluation.

[0040] As shown in the figure, the present invention comprises the following steps:

[0041] S1. Obtain online formula data in the CROHME dataset, and obtain the category and position of each character in the online formula data.

[0042] The design and application of the CROHME dataset has played a key role in the field of mathematical formula recognition, especially in promoting research and development in this field. The CROHME dataset is unique in that it provides data in two formats: online and offline. The information of the online data contains character positions and category labels, which makes it possible to enhance the data. Over time, in order to expand the scope of application and adapt to more research needs, the CROHME dataset has been expanded to include offline data, that is, images of formulas that have been written. These images do not contain information about the dynamic process of the strokes, but provide static formula images. Since the CROHME online dataset is annotated with character positions and categories, the category and position of each character can be obtained, which provides the possibility of replacing and deforming individual characters in the entire formula.

[0043] S2. Perform data preprocessing on samples in the online formula data.

[0044] In order to expand the application scope of the CROHME dataset and adapt to more research needs, it is necessary to preprocess the samples, including data noise reduction and data normalization. Among them, data noise reduction includes: deleting closely spaced coordinate points and points densely distributed along a straight line. Specifically, due to changes in users' writing habits, differences in the sampling frequency of trajectory data on different devices, and changes in device size, the training data contains a large amount of noisy data. The noise reduction method used in this example is that if a coordinate point is so close to its adjacent points that it is below a certain threshold, then the coordinate point is considered redundant.

[0045]

[0046] in,% d"s$ is the distance threshold. In this example, it is set to 0.01*'a"((,h), where w and h represent the width and height of the mathematical formula, respectively. That is, the threshold is not an absolute value but a relative value.

[0047] Noise reduction is a key step to ensure data quality and improve analysis accuracy. In addition to removing closely spaced coordinate points, which may be duplicate or redundant data due to measurement errors or data recording errors, attention should also be paid to points densely distributed along a straight line. These points may represent oversampling during data collection or continuous recording when the sensor moves on a straight path. The corresponding criteria for this judgment are as follows:

[0048]

[0049] in% cos Represents the angle cosine threshold, which is set to 0.99, that is, if the angle between two adjacent line segments is very small, they are judged to be redundant points.

[0050] In addition, given that different mathematical formulas are written by different people on various devices, the coordinate ranges of the mathematical formulas are different, so they need to be normalized to a consistent coordinate range. Data normalization includes: calculating the projection center of the coordinate point on the horizontal and vertical axes; calculating the standard deviation of the projection center; and normalizing the coordinate point based on the projection center and the standard deviation. Specifically, the first step is to calculate the projection center of the coordinate point on the horizontal and vertical axes:

[0051]

[0052] Among them: Ω represents the set of all line segments in the mathematical formula,

[0053] represents the length of the line segment L formed by (x1, y1) and (x2, y2),

[0054] p"(L) and p$(L) represent the projection lengths of line segment L on the x-axis and y-axis, respectively, namely:

[0055]

[0056] Then calculate the standard deviation of the projection center,

[0057] d x (L) = ∫ L (x-μ x ) 2 dL,

[0058] Get the standard deviation on the x-axis,

[0059]

[0060] After obtaining the projection center and standard deviation, the coordinate points can be normalized:

[0061]

[0062] The standard deviation on the x-axis is used to keep the aspect ratio of the mathematical formula unchanged. After denoising and normalizing the dataset, the data augmentation method can be used to expand the dataset.

[0063] S3. judge each character in the online formula data and replace the characters.

[0064] For the input online formula data of the CROHME dataset, each character of the formula is judged. If it is a fraction separator, it is put into the fraction array to swap the upper and lower parts. If it is a normal character, it is judged whether it is in the replaceable character group. If it is replaceable, the character is replaced in the same character group with a probability of 0.25, and then all characters are randomly combined and changed from several types of shearing, scaling, rotation and perspective transformation. After processing all the characters in the formula, the upper and lower contents of the fraction are swapped, that is, the upper and lower parts of the fraction are determined by the left and right boundaries of the fraction symbol, and then exchanged.

[0065] Among them, character replacement involves replacing existing characters in the formula with other character samples, thereby generating new formula samples with different semantic information from the original formula image. In the CROHME training set, there are 101 types of characters, totaling 85,802 characters, and each character is individually annotated in the online CROHME dataset. Since each character is essentially composed of multiple coordinate points, character replacement can be achieved. Since the goal of data augmentation in this paper is to expand the handwritten mathematical formula dataset, that is, to generate and simulate the effect of handwritten mathematical formulas through existing datasets, the overall semantics of mathematical formulas should be considered when doing data augmentation, that is, after data augmentation, the formula is still a legal formula. However, since the characters in the formula have different shapes and writing methods, such as the character "a" is a centered structural format, "g" is a lower structural format, and "f" is an upper structural format. Although they are all English letters, they cannot be directly replaced without accurate position information to avoid character overlap. In addition, some characters in the formula have specific semantics, such as brackets such as "(" and ")", integrals, summations, trigonometric functions, etc. These characters cannot be replaced with other characters, otherwise excessive semantic transformation will occur, resulting in errors in the formula itself. In this case, it is necessary to classify the characters. When replacing characters, only samples belonging to the same category are allowed to replace each other, such as Figure 2 shown.

[0066] S4. Scale and rotate the entire formula after character replacement, and update the LaTeX sequence corresponding to the current formula.

[0067] The present invention is a data enhancement method based on a character annotation dataset, which makes full use of the characteristics of the CROHME online dataset with character position and category annotations. First, the CROHME data is subjected to noise reduction and normalization, the noise of the data is removed, the influence of the model on the error information is reduced, the model is easier to learn and extract features, and the subsequent character replacement and the exchange of the upper and lower contents of the fraction structure are also convenient. The present invention performs random replacement of the same type of characters in the existing formula, and exchanges the upper and lower contents when encountering the fraction structure, changes the semantic information of the formula, and expands the diversity of the semantics of the data sample; deforms the local characters and the global formula, simulates the different styles of handwritten formulas, and helps to improve the generalization ability of the model; normalizes the LaTeX tags of the formula image, removes useless information, and avoids the situation where the same feature corresponds to multiple outputs.

[0068] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data enhancement method for a mathematical formula recognition model, characterized in that: The steps include: S1. Obtain online formula data in the CROHME dataset, and obtain the category and position of each character in the online formula data; S2. Performing data preprocessing on samples in the online formula data; S3. Determine each character in the online formula data and replace the characters; S4. Scale and rotate the entire formula after character replacement, and update the LaTeX sequence corresponding to the current formula.

2. According to claim 1, a data enhancement method for a mathematical formula recognition model is characterized in that: In step S2, the data preprocessing includes data noise reduction and data normalization.

3. According to claim 2, a data enhancement method for a mathematical formula recognition model is characterized in that: In step S2, the data noise reduction includes: deleting closely spaced coordinate points and points densely distributed along a straight line.

4. A data enhancement method for a mathematical formula recognition model according to claim 2 or 3, characterized in that: In step S2, the data normalization includes: calculating the projection center of the coordinate point on the horizontal and vertical axes; calculating the standard deviation of the projection center; and normalizing the coordinate point based on the projection center and the standard deviation.

5. According to claim 4, a data enhancement method for a mathematical formula recognition model is characterized in that: In step S2, the projection center of the coordinate point on the horizontal and vertical axes is calculated. Among them: Ω represents the set of all line segments in the mathematical formula, represents the length of the line segment L formed by (x1, y1) and (x2, y2), p"(#) and p$(#) represent the projection lengths of line segment L on the x-axis and y-axis, respectively, namely:

6. According to claim 5, a data enhancement method for a mathematical formula recognition model is characterized in that: In step S2, the standard deviation of the projection center is calculated. Get the standard deviation on the x-axis, 7. The data enhancement method for a mathematical formula recognition model according to claim 6, characterized in that: In step S2, the coordinate points are normalized based on the projection center and the standard deviation, where:

8. A data enhancement method for a mathematical formula recognition model according to claim 1 or 2, characterized in that: In step S3, if it is a fraction separator, it is put into the fraction array to swap the contents of the upper and lower parts; if it is a character in the replaceable character group, the character is replaced.

9. The method for data enhancement for a mathematical formula recognition model according to claim 8, characterized in that: In step S3, the replaceable character groups are classified, the character replacement is performed in the same character group, and then all the characters are randomly combined and changed from multiple types of shearing, scaling, rotation and perspective transformation.

Citation Information

Patent Citations

  • Handwritten sample recognition method and system based on sample enhancement

    CN109858488A

  • Method for reducing over-fitting of network training during offline handwritten mathematical formula recognition

    CN110084239A

  • Mathematical formula identification method based on operator action range and center line

    CN113139548A

Cited By

  • Graph structure guidance-based stylized handwritten formula generation method

    CN119887966A

  • Mathematical formula processing method and device, computer equipment and readable storage medium

    CN120449867A

  • Handwriting answer recognition method based on visual language model

    CN121527791A