Computation question image recognition and automatic discrimination method based on deep learning

Abstract

The invention discloses a calculation question image recognition and automatic discrimination method based on deep learning, relates to the technical field of character recognition and deep learning in OCR (Optical Character Recognition), and solves the problem of low calculation question recognition accuracy caused by difficulty in recognition of image characters and two-dimensional structures of handwritten mathematical calculation questions of pupils in the prior art. The method comprises the following steps: firstly, carrying out data processing on an equation-containing image, and dividing the processed image data into a training set and a test set; and constructing a deep learning model of an encoder and decoder structure with a counting module, training the model by using the training set, and then calling the trained learning model to identify a computational question equation to complete automatic correction of exercises. Finally, the automatic identification and correction of the mathematical homework of the pupils with powerful functions can be realized. According to the method, aiming at the problem of inaccurate attention, a counting module is added, so that the attention is more accurate, and meanwhile, a counting result can represent the number of symbols and serve as additional global information to improve the recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of OCR Chinese character recognition and deep learning technology, and in particular to a method for recognizing and automatically distinguishing a calculation problem image based on deep learning. Background Art

[0002] Due to the application of deep learning in recent years, optical character recognition has achieved rapid development and has been widely used in smart office, teaching assistance and industrial inspection. However, in the education industry, there is still a large gap in the market for course tutoring tools for teaching. Whether on campus or at home, grading students' homework is still a big burden for parents and teachers. There are still some challenges in realizing intelligent grading of homework: first, due to the inconsistency of shooting and lighting conditions, the target characters in the photos are tilted and blurred; then, in the process of string recognition, due to the different habits of different people in writing symbols in calculation problems, the shapes of symbols written by students are diverse, which will increase the difficulty of recognition for the model to a certain extent. To accurately realize the image recognition and classification of four arithmetic formulas (long character strings) of calculation problems, it is essential to extract more expressive features; finally, after the recognition and classification are completed, due to the presence of multiple operation symbols in the test questions, it is also very important to correctly divide the formula types, retrieve and verify the answers, and output feedback correction results.

[0003] The existing patent document, publication number CN1113435441A, publication date 2021.09.24, proposes a method for recognizing and automatically correcting images of arithmetic formulas, which aims to use a deep learning network model that combines a convolutional neural network and a bidirectional long short-term memory artificial neural network to recognize segmented images. However, it can only recognize and automatically correct simple arithmetic formula images, and it is difficult to correct images such as fractional formulas and decimal formulas. Moreover, it is difficult to accurately recognize the characters in the formula image.

[0004] However, in the prior art, there is still no good solution to the problems of disordered order and unclear handwriting that are inevitably present in mathematical formulas in real scenarios. Based on the problems that the current recognition model has low recognition accuracy of mathematical characters, the two-dimensional structure between mathematical characters is difficult to distinguish, and the method of grading homework relies on the network connection and the recognition of handwritten numbers has its own characteristics, the present invention proposes a method for image recognition and automatic discrimination of calculation problems based on deep learning. Summary of the invention

[0005] The present invention aims to solve the problem in the prior art that it is difficult to recognize image characters and two-dimensional structures of handwritten math problems for primary school students, resulting in a low recognition accuracy of math problems. A method for image recognition and automatic discrimination of math problems based on deep learning is provided.

[0006] A method for recognizing and automatically distinguishing computational problems based on deep learning is implemented by the following steps:

[0007] Step 1: Obtain a computational problem image dataset, and divide the computational problem image dataset into a training set and a test set;

[0008] Step 2: construct a deep learning network model with an encoder and decoder structure of a counting module, and train the deep learning network model using the training set to obtain a trained deep learning network module;

[0009] The deep learning network model uses a convolutional neural network as an encoder to extract feature maps from the calculation problem image, uses a symbol counting module to add count vectors to the feature maps, and uses an LSTM recurrent neural network as a decoder to gradually output predicted Latex formula recognition results;

[0010] Step 3: Use the test set to test the trained encoder and decoder models, output the Latex formula recognition results, and complete the recognition and automatic identification of calculation problem images.

[0011] Beneficial effects of the present invention: The method of the present invention first processes the data of the image containing the mathematical formula to obtain a training set. A deep learning model with an encoder and decoder structure of a counting module is constructed and trained, and then the trained recognition model is called to recognize the mathematical formula of the calculation problem and complete the automatic correction of the exercises. Finally, a powerful automatic recognition and correction of elementary school students' mathematics homework can be realized, which has the following advantages:

[0012] 1. To address the problem of inaccurate attention, the method of the present invention proposes to make attention more accurate by adding a counting module. At the same time, the counting result can represent the number of symbols and can be used as additional global information to improve recognition accuracy.

[0013] 2. The method of the present invention proposes a data enhancement method, which increases the diversity of data through image processing and random rotation methods, enhances the features of the temporal aspect and effectively models the contextual information in the formula image.

[0014] 3. The method of the present invention proposes a variety of optimization strategies, such as hyperparameter adjustment strategy and learning rate decay strategy, to further improve the accuracy of model recognition without increasing the model size. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the method for image recognition and automatic discrimination of calculation problems based on deep learning according to the present invention;

[0016] Figure 2The schematic diagram of the image recognition model for the computational problem.

[0017] Figure 3 This is a schematic diagram of the image recognition results of the computational problem based on deep learning described in the present invention. DETAILED DESCRIPTION

[0018] Combination Figures 1 to 3 The present embodiment is described as follows: a method for recognizing and automatically distinguishing a computing problem image based on deep learning, which is implemented by the following steps:

[0019] Step S1, preparation of calculation problem image dataset: collect calculation problem images of primary school students' mathematics homework to construct a handwritten formula dataset, and preprocess the handwritten formula dataset; the specific process is as follows:

[0020] Step S11: Use a high-resolution scanner to scan the student test papers and calculation homework, collect the calculation problem images, and ensure that the images are clear and without distortion. Then crop and adjust the images to keep only the calculation problem images;

[0021] Step S12: compare hash values ​​of the computational problem images retained in step S1, detect and delete images with the same hash values, and manually check the images: manually browse the image set and delete duplicate images.

[0022] Obtaining a detected computational problem image;

[0023] Step S13, performing a data enhancement operation on the detected calculation problem image obtained in step S12, generating more diverse data by randomly rotating the image by 0 to 15 degrees, and obtaining a calculation problem image after data enhancement;

[0024] Step S14, image annotation, manually annotate the computational problem image obtained in step S13, obtain the corresponding Latex code label, and complete the computational problem image data set. The computational problem image data is divided into a training set and a test set according to a ratio of 9:1. The training set is used for training in subsequent steps, and the test set is used to perform performance testing on the trained model.

[0025] In this embodiment, the calculation problem image data set includes original handwritten mathematical formula images and corresponding Latex code labels; specifically, the maximum label length of the label setting is 100. This effectively reduces the amount of calculation in the model training process and improves the training speed.

[0026] Step S2, construct a deep learning network model with an encoder and decoder structure having a counting module; input the training set into the deep learning network model to identify the Latex formula of the calculation problem image. The deep learning network model uses a convolutional neural network as an encoder to extract feature maps from the calculation problem image, then uses a symbol counting module to add additional global information to the feature map, and finally uses a long short-term memory (LSTM) recurrent neural network as a decoder to gradually output the predicted Latex formula.

[0027] like Figure 2 As shown, the deep learning network model workflow specifically includes:

[0028] Step S21: Input the computational image of size H×W into the encoder, first perform preliminary feature extraction through the basic convolution layer with a convolution kernel of 3×3 and a step size of 1, then sequentially pass through the maximum pooling layer, batch normalization, and ReLU activation layer to extract global information features and local information features. Then, four residual convolution layers are used to fuse the extracted global information features and local information features, and finally output a feature map F with 256 channels. The convolution kernel size of the last residual convolution layer is 3×3. H and W are the height and width of the input image, respectively.

[0029] Step S22, after obtaining a feature map F with an output channel number of 256, use a symbol counting module to extract symbol features from the feature map F; first, extract symbol features through a 5×5 convolution layer and a normalization layer, then perform feature enhancement through an average pooling layer and two linear layers, and after obtaining the feature-enhanced symbol feature map, use a 1×1 convolution to reduce the number of channels of the symbol feature map to 41 dimensions, where the number of feature map channels represents the types of all symbols in the calculation problem image, and finally use a Sigmoid activation function to compress each element value in the feature map with symbol feature enhancement to the range of (0,1) to obtain a symbol density feature map. Then, after adding all elements of each dimension in the symbol density feature map, a 41-dimensional count vector V can be obtained, and each value in the count vector is the count value of the predicted corresponding symbol.

[0030] Step S23: According to the feature graph F obtained in step S21 and the count vector V obtained in step S22, the 256-dimensional feature graph F and the 41-dimensional count vector V are matched in size, and the dimension splicing operation is performed to obtain the feature graph F of the fused count vector. The feature graph F of the fused count vector is split according to the dimension to obtain a feature vector sequence C represents the dimension of the new feature map, C=256.

[0031] Step S24: The feature vector sequence The vector f in i In the input decoder one by one, the decoder uses a long short-term memory (LSTM) network to iteratively predict the output symbol y i , until the end symbol "end" is predicted, and the Latex prediction sequence of the calculation problem image is obtained The expression is where y i The symbol for the prediction at step i.

[0032] Step S3, using the test set to test the performance of the trained deep learning network model with encoder and decoder structure of the counting module, complete the hyperparameter adjustment of the hybrid neural network (learning rate, optimizer, batch size, random killing of neurons) to obtain the best performance on the test set. By adjusting the hyperparameters, ensure that the accuracy of the model on the test set tends to the optimal level when fitting, and finally complete the performance indicator evaluation of the deep learning network model with encoder and decoder structure of the counting module.

[0033] Step S4: Use the trained encoder and decoder models to recognize the student calculation problem images and output the Latex recognition results.

[0034] In this implementation, for the operation symbols in the Latex formula recognition result, the operation symbols include an equal sign, a greater than sign, and an equal sign, and the formula before the operation symbol and the formula after the operation symbol are saved respectively;

[0035] Retrieving the operation symbol in the formula, and selecting the corresponding calculation logic to perform operation and solve the formula;

[0036] Compare the results of the two types of operations. If the results of both operations conform to the logic of the operation symbols, the output answer is correct, such as Figure 3 As shown, the question is marked with a green frame, otherwise the output answer calculation error or the recognition abnormal feedback result is marked with a red frame.

[0037] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0038] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for image recognition and automatic discrimination of computational problems based on deep learning, characterized by: The method is implemented by the following steps: Step 1: Obtain a computational problem image dataset, and divide the computational problem image dataset into a training set and a test set; Step 2: construct a deep learning network model with an encoder and decoder structure of a counting module, and train the deep learning network model using the training set to obtain a trained deep learning network module; The deep learning network model uses a convolutional neural network as an encoder to extract feature maps from the calculation problem image, uses a symbol counting module to add count vectors to the feature maps, and uses an LSTM recurrent neural network as a decoder to gradually output predicted Latex formula recognition results; Step 3: Use the test set to test the trained encoder and decoder models, output the Latex formula recognition results, and complete the recognition and automatic identification of calculation problem images.

2. The method for image recognition and automatic discrimination of calculation problems based on deep learning according to claim 1, characterized in that: The specific process of step one is: Step 1: Scan student test papers and calculation homework using a high-resolution scanner to collect calculation homework images; Step 12: Compare the hash values ​​of the calculation problem images described in step 11, detect and delete images with the same hash value, and manually check the images to obtain the detected calculation problem images; Step 13: Perform data enhancement operation on the detected computing problem image obtained in step 22, and obtain the computing problem image after data enhancement by randomly rotating the image by 0 to 15 degrees; Step 14: Manually annotate the calculation problem images obtained in step 13, obtain the corresponding Latex code labels, and complete the preparation of the calculation problem image dataset.

3. The method for image recognition and automatic discrimination of calculation problems based on deep learning according to claim 1, characterized in that: In step 2, the specific process of using the training set to train the deep learning network model is: Step 2:

1. Input the image of the problem with a size of H×W into the encoder, where H and W are the height and width of the input image respectively; firstly, perform preliminary feature extraction through a basic convolution layer with a convolution kernel of 3×3 and a step size of 1, and then successively extract global information features and local information features through a maximum pooling layer, batch normalization processing, and a ReLU activation layer; then perform feature fusion on the extracted global information features and local information features through four residual convolution layers, and finally output a feature map F with a channel number of 256; Step 22: using a symbol counting module to extract symbol features from the feature graph F; First, the symbol features are extracted through a 5×5 convolution layer and a normalization layer. Then, the features are enhanced through an average pooling layer and two linear layers. A 1×1 convolution is used to reduce the number of channels of the symbol feature map to 41 dimensions. The number of channels of the feature map represents the types of all symbols in the calculation problem image. Finally, the Sigmoid activation function is used to compress each element value in the feature map obtained by symbol feature enhancement to the range of (0,1), and finally the symbol density feature map is obtained. Then, all elements of each dimension in the symbol density feature map are summed up to obtain a 41-dimensional count vector V, each value in the count vector V is a count value of the predicted corresponding symbol; Step 23: Match the size of the feature map F obtained in step 21 and the count vector V obtained in step 22 and perform dimension splicing operation to obtain a feature map of the fused count vector; split the feature map of the fused count vector according to the dimension to obtain a feature vector sequence C is the dimension of the feature map; Step 24: Sequence the feature vector The vector f in i Input into the decoder one by one, and the decoder iterates to predict the output symbol y i , until the end symbol end is predicted, and the Latex prediction sequence of the calculation problem image is obtained.

4. The method for image recognition and automatic discrimination of calculation problems based on deep learning according to claim 1, characterized in that: In step three, the operation symbols in the Latex formula recognition result, which include equal sign, greater than sign, equal sign, respectively save the formula before the operation symbol and the formula after the operation symbol; retrieve the operation symbol in the formula, select the corresponding calculation logic to calculate and solve the formula; compare the results of the two types of calculation solutions, if the results of both are consistent with the logic of the operation symbol, output the correct answer, otherwise output the feedback result of the answer calculation error or recognition abnormality.

Images