Primary school student mathematical equation recognition and automatic correction method based on deep learning

Abstract

The application discloses a kind of pupil mathematical formula recognition and automatic correction method based on deep learning, first, the image containing formula is preprocessed, and redundant frame information is filtered out, second, BTTR model based on data augmentation training is used to recognize formula, finally, the recognition result is corrected using the correction template covering multiple mathematical question types, and the automatic correction of multiple question types is completed.The application realizes the process from mathematical formula input to correction result output, the sequence model based on Transformer can solve the problem of character level noise and long sequence dependence during recognition, the training method combined with data augmentation effectively improves the generalization ability of the model, and the diversified template can match different types of formula, finally, a powerful automatic recognition and correction method for pupil mathematical homework is realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of character recognition in OCR, deep learning and template matching, and mainly relates to a pupil mathematical formula recognition and automatic correction method based on deep learning. BACKGROUND

[0002] Optical Character Recognition (OCR) refers to the process of analyzing and recognizing image files of text materials to obtain text and layout information, that is, recognizing the text in the image and returning in the form of text. According to the recognition scene, OCR can be roughly divided into special-purpose OCR for recognizing specific scenes and general-purpose OCR for recognizing multiple scenes. Special-purpose OCR means focusing only on specific fields, while general-purpose OCR can be used in more complex scenes. According to the content of the recognized image, the scene can be divided into simple scenes with clear and fixed patterns and more complex natural scenes. The difficulty of natural scene text recognition is extremely high, and traditional text recognition techniques such as template matching have difficulty in dealing with low brightness, low contrast, uneven lighting, perspective distortion and missing occlusion. With the advent of the deep learning era, more people have begun to explore how to solve the difficulties of text recognition in natural scenes in the field of deep learning. In the Latex formula recognition, Yuntian Deng, Jeffrey Ling and Alexander M. Rush, et al. Image-to-Markup Generation with Coarse-to-Fine Attention [C]. / / Proceeding of International Conference on Machine Learning (ICML),

[0003] 2017:980-989. One article introduces CNN to extract picture features, and uses RNN to decode each row of the feature map to construct a visual compiler, which has achieved certain effect in recognition accuracy. Zhang Jiansu JunDu, et, al. Watch, Attend and Parse. An End-to-End Neural Network based Approach to Handwritten Mathematical Expression Recognition [J]. Pattern Recognition, 2017 (71): 196-206. One article improves the Encoder-Decoder framework, which does not use the traditional tree structure to parse the Latex formula, but uses FCN as the Encoder to extract picture information, selects GRU as the Decoder, and introduces Attention to better capture useful information for decoding. Then Zhang Janshu, Jun Du and Lirong Dai. Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition [C]. / / Proceedings of International Conference on Pattern Recognition (ICPR), 2018: 2245-2250. One article further improves the number of layers of FCN, introduces Multi-scale, and upgrades the Encoder to DenseNet, which further improves the recognition accuracy. After that, Liangcai Gao, Zhao Wenqi and Yan Zuoyu, et al. Handwritten Mathematical Expression Recognition with Bidirectionally Trained Transformer [C]. / / Proceedings of International Conference on Document Analysis and Recognition (ICDAR). 2021: 570-584. Introduces Transformer as Decoder and adopts bidirectional decoding to propose BTTR (Bidirectionally Trained Transformer), which greatly improves the recognition accuracy.

[0004] However, the existing model is mostly applied to latex formula picture analysis, and there are few studies on primary school mathematics homework recognition. Meanwhile, the difference between latex formula recognition lies in the diversity of question types, the difference in picture illumination, and the great influence of noise, which poses greater challenges to recognition. Although there are related products on the market, their accuracy in analyzing long formulas and vertical formulas is poor, and it is difficult to filter character noise. SUMMARY

[0005] The present application is aimed at the difficulties in the field of primary school handwritten mathematical formula recognition and the lack of correction paradigm in the prior art. A primary school mathematical formula recognition and automatic correction method based on deep learning is provided. First, the image containing the formula is preprocessed to filter out redundant frame information. Second, the BTTR model based on data augmentation training is used to recognize the formula. Finally, the recognition result is corrected using the correction template covering multiple mathematical question types, and the automatic correction of multiple question types is completed. The present application realizes the process from mathematical formula input to correction result output. The sequence model based on Transformer can solve the problems of character-level noise and long sequence dependence in recognition. The training method combined with data augmentation effectively improves the generalization ability of the model. The diversified template can match different types of formulas, and finally a powerful method of automatic recognition and correction of primary school mathematics homework is realized.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is: a primary school mathematical formula recognition and automatic correction method based on deep learning, characterized in that it comprises the following steps:

[0007] S1, image acquisition: determine the primary school mathematics homework formula image for training;

[0008] S2, image preprocessing: adopt a mathematical formula frame filtering algorithm to filter out redundant information in the image that is irrelevant to the formula, wherein the redundant information at least includes the frame of the mathematical formula and information other than mathematical characters or symbols;

[0009] S3, formula sequence acquisition: use the sequence model based on Transformer trained in combination with data augmentation to recognize the formula, and the step further comprises:

[0010] S31: given a sample, generate a random number, and if the random number is less than a predetermined probability value, perform Gaussian blur, add Gaussian noise, and color jitter on the sample;

[0011] S32: obtain the feature map of the sample based on the sequence model based on Transformer, and after position encoding the image, flatten the feature map into a one-dimensional vector according to the width and height, and complete the encoding process;

[0012] S33: decode the encoded feature vector to generate an equation sequence;

[0013] S4, automatic correction: for the equation sequence identified in step S3, a diversified template is constructed, and automatic correction is performed, the template at least including simple equations, inequalities, equations with units, fraction equations, detached equations, monomial equations, approximately equal equations, vertical equations and equations with remainders or repeating decimal equations.

[0014] Compared with the prior art, the present application has the beneficial effects:

[0015] (1) The sequence model BTTR based on the Transformer is applied in the equation recognition of the primary school mathematics homework, the equation is preprocessed by filtering the frame to reduce the redundant information irrelevant to the sample, the data augmentation technology is used to increase the Gaussian blur, Gaussian noise and color jitter operation on the sample to prevent the model from overfitting, and the model recognition accuracy is improved, and the problems of inaccurate long equation recognition and character noise in the mathematics homework are solved.

[0016] (2) The automatic correction equation template is constructed, a complete correction logic covering various equations is summarized, and equations that are difficult to process before, such as inequalities, vertical calculation, remainder calculation, monomial equation solving, detached calculation and unit conversion, can be processed, the application range of automatic correction is widened, and the automatic correction function of multiple equations is completed.

[0017] (3) The equation recognition can be directly performed on the new primary school mathematics homework image, and the correction result is returned, the operation is simple and convenient, the practicality is strong, and it is more in line with the actual demand. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is the flow chart of the primary school mathematics equation recognition and automatic correction method of the present application based on deep learning;

[0019] Figure 2 is the front and back comparison diagram of step S2 of the present application filtering the equation frame;

[0020] Figure 3 is the BTTR model framework diagram of the present application;

[0021] Figure 4 is the simple equation recognition and correction result diagram of the present application;

[0022] Figure 5 is the inequality recognition and correction result diagram of the present application;

[0023] Figure 6 is the equation with unit recognition and correction result diagram of the present application;

[0024] Figure 7is the fraction formula recognition and correction result graph of the present application;

[0025] Figure 8 is the fraction formula recognition and correction result graph of the present application;

[0026] Figure 9 is the one-variable one-degree equation recognition and correction result graph of the present application;

[0027] Figure 10 is the approximately equal formula recognition and correction result graph of the present application;

[0028] Figure 11 is the vertical formula recognition and correction result graph of the present application;

[0029] Figure 12 is the remainder formula recognition and correction result graph of the present application. DETAILED DESCRIPTION

[0030] The present application will be further illustrated below in conjunction with the drawings and specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the present application and not to limit the scope of the present application.

[0031] Example 1

[0032] Before describing the specific steps of the method, first give the relevant definitions and representations:

[0033] (a) Sample: a group of 75000 primary school mathematics homework formulas, cut from the public data set AEC-5k

[0034] (b) Class label: that is, the correct mathematical formula corresponding to a sample;

[0035] (c) Transformer-based sequence model (BTTR): Bidirectionally Trained Transformer

[0036] (d) Contour approximation (DP): Douglas-Peucker algorithm;

[0037] (e) Edge detection (Canny operator): Canny detection algorithm;

[0038] A primary school mathematics formula recognition and automatic correction method based on deep learning, mainly for the mathematical formulas in primary school mathematics homework, to identify and correct the return results, specifically including the following steps:

[0039] S1, image acquisition: determine the primary school mathematics homework formula image for training;

[0040] S2, image preprocessing: using a mathematical formula border filtering algorithm to filter out redundant information in the image that is irrelevant to the formula, the redundant information at least including the border of the mathematical formula and information other than mathematical characters or symbols;

[0041] The mathematical formula border usually exists in large quantities in primary school mathematics homework, and is characterized by: ① the area filled with mathematical characters or symbols; ② non-fixed shape, mainly rounded rectangle or square cornered rectangle; the filtering of redundant information in the image in this embodiment specifically includes:

[0042] S21, extracting the contour of the formula image: performing binaryzation processing on a given primary school mathematics homework formula color image, filtering with a 3*3 Gaussian convolution kernel, using a Sobel operator to calculate the image gradient and direction, performing non-maximum suppression on the eight directions of the pixel points, and using the average value and maximum value of the image pixel points as double thresholds to extract the image edge.

[0043] S22, identifying the edge contour shape: using the DP algorithm to fit the contour with the number of straight line segments, identifying the contour shape with the number of corner points, and preliminarily screening the formula border to be identified, and increasing the minimum circumscribed quadrilateral width threshold to limit the filtering of irrelevant borders.

[0044] S23, covering the image formula border pixels: covering the pixel value of the border with the median of the image pixel value to obtain the mathematical formula after filtering the border.

[0045] After a large number of experiments, when identifying the edge contour shape, the distance threshold of the DP algorithm fitting the contour should be set to 3, the corner point number interval should be set to 4-7, and the minimum circumscribed quadrilateral width threshold should be 0.8 of the image width, at this time, the effect of filtering the border is best, and the effect of the border filtering algorithm is as shown in Figure 2 Figure 2 That is, the before-and-after comparison of the image after step S2 processing.

[0046] S3, formula sequence acquisition: using a sequence model based on Transformer trained by combining data augmentation to recognize the formula, the step specifically comprising:

[0047] S31: given a sample, generate a random number, if the random number is less than a preset probability value, perform Gaussian blur, increase Gaussian noise, and color jittering on the sample. The reason why random rotation and random cropping are not adopted is that the sample label is not fine-grained enough, and cropping or rotating cannot be reflected on the label, which leads to the model being unable to infer the cropped part, and is not conducive to the model learning the semantic information of the image.

[0048] ​S32: A sequence model based on a Transformer obtains a feature map of the sample, and after position encoding of the image, the feature map is unfolded into a one-dimensional vector according to the width and height, and the encoding process is completed; the sequence model based on the Transformer BTTR framework is as shown in Figure 3 , and the execution mode is as follows:

[0049] S321: Obtain a sample feature map. Let I l be the output of the lth DenseBlock, let I1 be the output of the input first DenseBlock, I2 be the output of I1, …, and let I l-1 be the output of I l-2 . H l is a function composed of three consecutive layers (BN, RELU, 3*3Conv), and the DenseBlocks are connected through a Transition, let T l be the Transition layer in the (l-1)th DenseBlock and the lth DenseBlock, which is a function composed of two consecutive layers (1*1Conv, 2*2Avgpooling) and is obtained by the following formula I l .

[0050] I l = T l (H l ([I0;I1;…;I l-1 ])

[0051] According to the above formula, the output of the last layer is obtained, that is, the feature map f of the image.

[0052] S322: The image is position encoded. The following formula uses sinusoidal positional encoding in the Transformer, and the process is as shown in the following formula:

[0053]

[0054]

[0055] H, W are the height and width of the image respectively, and the feature map encoding process is as shown in the following formula:

[0056]

[0057]

[0058] S323: The feature map is unfolded into a one-dimensional vector x according to the width and height, and the encoding process is completed.

[0059] S33: BTTR adopts Transformer as Decoder to decode the encoded feature vector to generate the formula sequence; the sample label and the feature Figure 1 and input the Decoder module in Transformer, generate the mathematical formula sequence by using the autoregressive decoding method, and in the application, there is no label input, and the autoregressive decoding and Beamsearch method are used until the stop identifier is generated. The specific operation is as follows:

[0060] (a) In the training, the bidirectional decoding calculation loss function is adopted, for a target formula sequence y target = {y1,..., y T}, the corresponding bidirectional target sequence is as follows

[0061]

[0062]

[0063] Let be the forward sequence generated by Transformer in the training, and the generation method is as follows:

[0064]

[0065] Where x is the sample graph feature vector, and θ is the model parameter.

[0066] Similarly, the reverse sequence can be generated according to the above formula, the CrossEntropy loss function is used to calculate the loss of the generated sequence and the target sequence respectively, and the parameter adjustment is performed by using the back propagation algorithm.

[0067] (b) In the application, the autoregressive decoding and Beamsearch method are adopted to generate the mathematical formula sequence Since there is no label in the application, when the stop identifier <eos>"Time to stop decoding. The decoding process is shown in the following formula:

[0068]

[0069] For different formula types, in order to distinguish, special symbols in the formula sequence are identified, such as using "!" to identify vertical formula, "#" to identify detached formula, "x" to identify monomial equation, "\\frac" to identify fraction, "……" to identify remainder, "…" to identify infinite repeating decimal, etc.

[0070] S4, automatic correction: for the formula sequence identified in step S3, a diversified template is constructed for automatic correction, and the template at least includes simple equation, inequality, formula with unit, fraction formula, detached formula, monomial equation, approximately equal formula, vertical formula, and formula with remainder or infinite repeating decimal.

[0071] Before correcting the formula, the following preprocessing is performed: ① extracting the identified formula equal sign or inequality sign "=, <, >" from the brackets; ② changing the square brackets to round brackets; ③ changing the proportion sign ":" to division sign " / ", and unifying the format. Then, pre-judgment is performed, specifically: ① judging whether there is a special empty character in the formula, if there is, it represents that the student has not filled in the answer, and it is judged as wrong; ② judging whether the brackets in the formula are closed, if the brackets are not closed, it is directly judged as wrong.

[0072] Template construction of simple equation: for simple equation, it has the following characteristics: the formula does not have unit or fraction identifier "\\frac" or variable x; the formula is a pure equation, does not contain inequality or approximately equal sign, and may have multiple equal signs. For example: "4+3+4=7+4=11". The specific judgment logic is: ① using the equal sign to separate the formula into independent formula; ② judging whether the formula is empty, if it is empty, it is directly judged as wrong, (the following operations of separating independent formula are not repeated) ; ③ converting the formula into postfix expression, and using expression evaluation algorithm to obtain the result of each formula; ④ comparing the results of each formula, if one of them is not equal, it is judged as wrong, and if all of them are consistent, it is judged as correct. Figure 4 The example is the recognition and correction result of simple equation.

[0073] Template construction of inequality: similarly, it needs to be consistent with the requirements of the above simple equation template construction, and the only difference is that it appears inequality instead of equal sign, for example: "4+4<9<10". The specific judgment logic is: ① using inequality to separate into independent formula; ② evaluating the formula, and judging according to the type of inequality, if there is a mathematical logic that does not meet the requirements, it is judged as wrong. Otherwise, if all of them meet the requirements, it is judged as correct. Figure 5 The example is the recognition and correction result of inequality.

[0074] Existence of unit formula template construction: such formula has a variety of conditions, such as unit conversion "1 square kilometer = 10000 square meters", the unit is wrapped in parentheses "25 (cm) * 22 (cm) = 556 (cm^2)", the unit before the operand is wrapped in parentheses "8100 m^2 = (0.81) hectares", multiple unit bracket operation "(7 yuan 5 jiao - (5 yuan - 1 jiao)) = 2 yuan 4 jiao", unit inequality operation "3000 grams < 1 ton < 2 tons". Specific judgment logic: ① build unit conversion dictionary in advance, such as in length unit: "m": "*1", "cm": "*0.01"; ② according to symbol separation into independent formula; ③ construct regular expression template to take out the unit wrapped in parentheses, for "cm^2" this kind of unit with "^", the corresponding regular expression is r"([(].*\^\d[)])", and the rest of the unit corresponding to the regular expression is r"(\({}\))", then locate the position of the parentheses and delete it; ④ construct regular expression to take out the operand wrapped in parentheses before the unit, r"([(][0-9]\d*\.?\d*[)]{})", where {} is filled in the corresponding unit, and for the same reason, match the parentheses and delete them; ⑤ according to the operator, cut the unit subsequence into multiple units, and convert the unit in the subsequence using the dictionary constructed in advance, if two or more units appear in the unit subsequence, it proves to be a multi-unit operation, which needs to be supplemented with a plus sign between the units, and wrapped in parentheses to maintain the operation order; ⑥ for each independent formula, operate ③④⑤; ⑦ judge the converted equivalent formula according to the logic of equality or inequality. Figure 6 Examples are the identification and correction results of the existence of unit formula.

[0075] Fraction formula template construction: for such formula, the feature is to have "\\frac" identifier. It can be divided into two cases: with fraction conversion and operation "\\frac{77}{72} > 1 \\frac{5}{12}", fraction operation "5 + ( \\frac{4}{5} + \\frac{5}{8} ) * 40 = 57". Specific judgment logic: ① according to symbol separation into independent formula; ② according to the fraction identifier "\\frac" to cut the independent formula, construct regular expression r"[{](.*?)[}]" to match the numerator numerator and denominator denominator; ③ convert the matched fraction expression to (numerator / denominator) form, add parentheses to ensure the rationality of the operation, and then combine the cut subsequence. ④ for each independent formula, operate ②③ to get the converted equivalent formula. ⑤ judge the converted equivalent formula according to the logic of equality and inequality. Figure 7 The example shows the recognition and grading results of fraction equations.

[0076] The construction of a multi-step equation template: A multi-step equation consists of multiple equal signs, with "#" indicating a newline. For example, "428+397+204#=825+204#=1029". Specific judgment logic: ① Replace "#" with an empty string to create a multi-step equation; ② Determine if the equation contains fractions or units. If so, use steps ②③④⑤ in the equation template construction for equations containing units, or steps ①②③④ in the fraction equation template construction to convert the equation containing units or fractions; ③ Judge the converted equation according to the judgment logic of the simple equation template construction. Figure 8 The example shows the recognition and correction results of the multiplication table.

[0077] Construction of a linear equation template in one variable: A linear equation in one variable has the following characteristics: it has an identifier variable "x", it has no units, it may contain fractions, such as "x=3*\\frac{7}{5}#x=\\frac{27}{5}", and the multiplication sign before the variable "x" is omitted, such as "4x+1400=11000#x=2400". The judgment logic is as follows: ① Separate the equations into multiple independent equations based on "#"; ② If fractions exist in the equations, convert them using steps ①②③④ in the fraction equation template construction; ③ In each equation, use regular expressions to match x and its preceding operands, and add the multiplication sign; ④ Solve each independent equation, find the corresponding value of x, and compare them. If all values ​​are the same, the intermediate process and the answer are correct, and the equation is judged as correct; otherwise, it is incorrect. Figure 9 The example shows the identification and correction results of a linear equation in one variable.

[0078] Construction of Approximate Equation Templates: Characteristics of approximate equations: Generally, there is only one approximate equal sign, such as "1185*22≈12000"; the two sides of the equation are not completely equal, such as "9.008≈9.01"; some contain fractions or unit operations, such as "\\frac{32}{30}*8≈240"; some also contain equal signs, such as "599800000=(5.99) billion≈(6) billion"; Specific judgment logic: ① Construct the sign change relationship of the symbols in advance, such as "+" corresponds to "-", "*" corresponds to " / "; ② If there are fractions or units in the equation, use the above method to convert them; ③ Divide the equation into left and right expressions according to the approximate equal sign; ④ Move all operators and operands in the right expression except for the first operand to the left expression. If there is an equal sign, they also need to be moved to the expression on the left side of the equal sign. The number of digits in the left-hand operation result is determined by the number of digits in the right-hand operand (if there is a decimal point, the number of digits after the decimal point is retained; otherwise, the number of digits before the decimal point is retained). ⑤ The retained results are compared. If they are all consistent, the result is considered correct; otherwise, it is considered incorrect. Figure 10 Example is approximately equal to the formula recognition and correction results.

[0079] Vertical formula template construction: vertical type includes addition and subtraction, multiplication and division, the vertical writing of addition and subtraction has similarity, such as "! 1000 # -! 778 # &!! 222", using "#" to separate, the first is the left operand, followed by the operator and the operand, the third is the result. The vertical of division is "!! 5 # 9 \ sqrt {45} #! 45 #!! 0", using "#" to separate, the first is the quotient, the second is the divisor and the dividend, the third is the intermediate process, and the final is the remainder. Each corresponds to a template. The judgment logic of the division vertical is: ① remove the "!" filler symbol, and separate it into multiple formulas according to "#"; ② perform the null operation, and if any element (quotient, divisor, dividend, remainder) is missing, it is judged as an error; ③ take the quotient, divisor, dividend, and remainder in order; ③ convert it into a new equation of "quotient * divisor + remainder = dividend", and judge it according to the judgment logic of the simple equation template construction. For non-division calculation, the specific judgment logic is: ① take out the "!" filler symbol and separate it into multiple formulas using "#"; ② judge null and take out the left operand, operator, right operand, and answer in order; ③ convert it into a simple equation form and judge it according to the simple equation template construction. Figure 11 Example is the recognition and correction results of vertical formula.

[0080] There is a remainder or wireless loop decimal formula template construction: there is a remainder, such as "(33) / (5)=5......8", the judgment logic is: ① separate the left formula and the right formula according to the equal sign; ② get the quotient, divisor, dividend, and remainder in order; ③ convert the formula to a simple equation to judge. There is an infinite loop decimal formula, such as "29 / 11=2.6363···", the judgment logic is: ① separate the left formula and the right formula, and remove the infinite loop identifier of the right formula; ② determine the number of bits to be retained in the result of the left formula according to the operand of the right formula, without rounding, but directly truncating; ③ compare the results of the left formula and the right formula, if they are consistent, judge as correct, otherwise, judge as error. Figure 12 It is the recognition and correction results of the formula with remainder.

[0081] Given a formula, after preprocessing and pre-judgment, the template matching order is: ① judge whether it is an approximately equal formula; ② judge whether it is a monomial equation; ③ judge whether it is a vertical formula; ④ judge whether it is a detached formula (recurrence equation); ⑤ judge whether it is a formula with units; ⑥ judge whether it is a fraction formula; ⑦ judge whether it is a formula with remainder or infinite loop decimal; ⑧ judge whether it is an inequality or a simple equation. The template matching of automatic correction is completed, and the judgment result of the formula is finally presented.

[0082] In order to verify the reliability and correctness of the method, the method and other models are compared, and the experimental results are shown in the following table:

[0083]

[0084] As can be seen from the above table, the existing primary school mathematics formula sequence recognition model on the market lags behind the model BTTR adopted by the present application in recognition accuracy, and after adding the filter frame and data augmentation technology, the recognition accuracy is further improved, proving the effectiveness of the method adopted by the present application.

[0085] The present application realizes the process from mathematical formula input to correction result output, the sequence model based on Transformer can solve the problems of character level noise and long sequence dependence in recognition, and the training method combined with data augmentation effectively improves the generalization ability of the model, and the diversified template can match different types of formulas, finally realizing a powerful primary school mathematics homework automatic recognition and correction method, which is simple and convenient to operate, practical, and more in line with actual needs.

[0086] It should be noted that the above content only illustrates the technical idea of the present application, and cannot limit the protection scope of the present application, and for ordinary skilled persons in the technical field, a number of improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements fall within the protection scope of the claims of the present application.< / eos>

Claims

1. A method for recognizing and automatically correcting mathematical expressions of pupils based on deep learning, characterized in that, The method comprises the following steps: S1, image acquisition: determining the pupil's math homework formula image for training; S2, image preprocessing: using a mathematical formula border filtering algorithm to filter out redundant information in the image that is irrelevant to the formula, the redundant information at least including the border of the mathematical formula and information other than mathematical characters or symbols; S3, formula sequence acquisition: using a sequence model based on Transformer trained in combination with data augmentation to recognize the formula, the step further comprising: S31: given a sample, generating a random number, if the random number is less than a preset probability value, performing Gaussian blur, adding Gaussian noise, and color jittering on the sample; S32: using a sequence model based on Transformer to obtain the feature map of the sample, and after position encoding of the image, flattening the feature map into a one-dimensional vector according to the width and height to complete the encoding process; S33: decoding the encoded feature vector to generate a formula sequence; S4, automatic correction: for the formula sequence recognized in step S3, constructing diversified templates for automatic correction, the templates at least including simple equations, inequalities, formulas with units, fraction formulas, detached formula, one-variable linear equations, approximately equal formulas, vertical formulas, and formulas with remainders or infinite repeating decimal.

2. The elementary school mathematics equation recognition and automatic correction method based on deep learning according to claim 1, wherein, The step S2 further comprises: S21, extracting the contour of the formula image: performing binaryzation processing on a given pupil's math homework formula color image, filtering with a 3*3 Gaussian convolution kernel, calculating the image gradient and direction using a Sobel operator, implementing non-maximum suppression on the pixel points in eight directions, and using the average value and maximum value of the image pixel points as double thresholds to extract the image edge; S22, identifying the edge contour shape: using the DP algorithm to fit the contour with the number of straight line segments, identifying the contour shape with the number of corner points, screening the formula border to be recognized, and further increasing the minimum circumscribed quadrilateral width threshold to limit the filtering of irrelevant borders; S23, covering the formula border pixels of the image: covering the pixel values of the border with the median of the image pixel values to obtain the mathematical formula after filtering out the border. 3.The elementary school student mathematical equation recognition and automatic correction method based on deep learning according to claim 2, characterized in that: In the step S22, when identifying the edge contour shape, the distance threshold of the DP algorithm fitting the contour is set to 3, the corner point number interval is set to 4-7, and the minimum circumscribed quadrilateral width threshold is 0.8 of the image width. 4.The elementary school student mathematical equation recognition and automatic correction method based on deep learning of claim 3, wherein, The step S32 further comprises: S321: Acquire the sample feature map: let I l be the output of the lth DenseBlock, I1 be the output of the input of the 1st DenseBlock, I2 be the output of I1, …, I l-1 be the output of I l-2 , H l be a function composed of three consecutive layers (BN, RELU, 3*3Conv), the DenseBlocks are connected through Transition, T l be the Transition layer in the (l-1)th DenseBlock and the lth DenseBlock, then the output of the last layer is obtained through the following formula, which is the feature map f of the image: I l = T l (H l ([I0; I1;...; In;...; In+1;...; In+M-1])) l-1 ])) S322, position encoding of the image: the process is shown in the following formula: wherein d is the dimension size, and pos is the current point position, is the vector with position information added to the current position pos of the sequence, i is the index of the current position encoding vector, 2i indicates that the current index is even, 2i+1 indicates that the current index is odd, and φ represents the current sequence. The feature map encoding process is shown in the following formula: wherein H, W are the height and width of the image respectively, represent the coordinate points after normalization, represent the encoding vectors after adding position information in the height dimension direction sequence, represent the encoding vectors after adding position information in the width dimension direction sequence, f x,y,d represent the feature map after position encoding of the image; S323, encoding: flattening the feature map into a one-dimensional vector x according to the width and height to complete the encoding process. 5.The elementary school student mathematical equation recognition and automatic correction method based on deep learning according to claim 4, wherein, In the step S33, a bidirectional decoding is taken to calculate the loss function during training, a CrossEntropy loss function is used to calculate the loss of each of the generated sequence and the target sequence, a back propagation algorithm is used to adjust the parameters, and an autoregressive decoding and a Beamsearch are used to generate the mathematical formula sequence during application The decoding process is shown in the following formula: where θ is the Decoder model parameter, y ≤i represents the generated equation sequence [BOS, y1, y2… y i ], y i+1 represents the newly generated character, T is the maximum length of the equation sequence, and BS represents the selection mode of the newly generated character y i+1 . 6.The elementary school student mathematical equation recognition and automatic correction method based on deep learning according to claim 5, wherein, Before the automatic correction in the step S4, it is judged whether there is a special empty character in the formula, if yes, it is directly judged as an error, representing that the pupil has not filled in the answer; It is further judged whether the brackets in the formula are closed, if not, the formula with unclosed brackets is directly judged as an error. 7.The elementary school student mathematical equation recognition and automatic correction method based on deep learning of claim 6, wherein, The step S4 automatically corrects the order of the template as follows: ① judging whether it is an approximately equal formula; ② judging whether it is a monomial linear equation; ③ judging whether it is a vertical formula; ④ judging whether it is a detached formula; ⑤ judging whether it is a formula with units; ⑥ judging whether it is a fraction formula; ⑦ judging whether it is a formula with a remainder or an infinite repeating decimal; and ⑧ judging whether it is an inequality or a simple equality.

Images