Intelligent Arithmetic Problem Grading System Based on MixNet-YOLOv3 and Convolutional Recurrent Neural Network (CRNN)

Abstract

The application discloses an arithmetic question intelligent marking system based on a MixNet-YOLOv3 and a convolution recurrent neural network CRNN (Convolutional Recurrent Neural Network), which can automatically identify the meaning of each arithmetic question in a test paper and make a judgment, so as to reduce the human and time costs brought by traditional manual marking of the test paper, thereby improving the teaching efficiency. The intelligent marking system mainly comprises two modules of detection and identification. In view of the characteristics of dense distribution and variable font of the arithmetic questions and the demand of network lightening, the detection module adopts a MixNet-YOLOv3 network of multi-scale semantic and positioning feature fusion to realize extraction of the frame information of the arithmetic questions; in the identification module, the arithmetic questions extracted by the previous module will pass through a convolution recurrent neural network CRNN network based on a connectionist temporal classification CTC (Connectionist Temporal Classification) decoding mechanism to obtain semantic information of the arithmetic questions; and finally, arithmetic logic operation is used to judge whether the purpose of each arithmetic question is correct or not. The system created by the application can undertake the arithmetic question marking task of a mobile device end.

Description

Technical Field

[0001] This invention relates to an intelligent grading system for arithmetic problems based on MixNet-YOLOv3 and convolutional recurrent neural networks (CRNN), belonging to the field of text detection and recognition at the intersection of computer vision and natural language processing. Background Technology

[0002] With advancements in technology and transformations in the education industry, the inefficiencies, quality issues, and lack of fairness inherent in traditional marking processes have become increasingly prominent, hindering the overall improvement of teaching standards. Meanwhile, the rapid development of digital image processing and computer vision technologies has drawn attention to intelligent marking, which can significantly improve marking efficiency, reduce teachers' workload, save parents' time, and optimize the allocation of educational resources.

[0003] Taking arithmetic problems for primary and secondary school students as an example, considering that arithmetic problems are generally composed of printed and handwritten text, the styles of printed text from publishers and the styles of handwritten text from students are constantly changing, and there is also a certain degree of alteration. Traditional optical character recognition (OCR) technology based on image processing and machine learning cannot meet the detection tasks of complex scenarios.

[0004] A typical OCR technology roadmap includes three parts: image preprocessing, text detection, and text recognition. The technical bottlenecks affecting recognition accuracy are text detection and text recognition, which are also the core of OCR technology. In traditional OCR technology, image preprocessing typically addresses image imaging issues, including geometric transformations, distortion correction, blur removal, image enhancement, and lighting correction. Text detection identifies the location, extent, and layout of text, often including layout analysis and line detection. Text detection primarily addresses where the text is and its extent. Text recognition, building upon text detection, identifies the text content, converting text information in the image into actual text. Text recognition primarily addresses what each character represents. Summary of the Invention

[0005] Purpose of the Invention: To address the aforementioned problems, this invention proposes an end-to-end arithmetic problem grading system. The system mainly covers two branches: detection and recognition. First, the YOLOv3 algorithm is used to detect the boundaries of each problem. Through weight allocation, the network can more easily learn the boundaries of the level, and the extraction network is replaced with a more lightweight MixNet network without sacrificing accuracy. In the recognition part, a convolutional recurrent neural network (CRNN) network that combines accuracy and efficiency is used. Through the combination of convolutional neural networks (CNN) and long short-term memory networks (LSTM), the network accurately learns the semantic information of the problem. Finally, arithmetic logic is used to determine whether the answer is correct and return the correct answer.

[0006] Technical solution: The implementation of the technical solution of this invention includes the following steps:

[0007] (1) The original detection dataset is preprocessed, and the generalization of the samples is expanded by data augmentation. At the same time, the K-means clustering algorithm is used to generate prior boxes that are adapted to the scenario for training the detection network.

[0008] (2) Input the entire image into the lightweight MixNet network model to perform image localization and semantic feature extraction, and obtain multi-scale features representing the global information of the image;

[0009] (3) The feature fusion technology of Feature Pyramid Network (FPN) is used to fuse features of different scales and feed them into the corresponding YOLOv3 prediction modules. The generated feature maps are calculated based on the two sets of prior boxes. The prediction is based on the best prior box. The loss function is calculated based on the predicted coordinates, confidence, category information and label information. A more accurate detection model is obtained through iterative training.

[0010] (4) Based on the bounding boxes of the arithmetic problems output by the detection network, the extracted arithmetic problems and the labeled text information are combined to form the dataset of the recognition module;

[0011] (5) The extracted arithmetic problem image is converted into a grayscale image and fed into the convolutional recurrent neural network (CRNN). It is necessary to first extract features through the convolutional neural network (CNN) to obtain feature maps, then convert them into sequences and feed them into the bidirectional long short-term memory network (LSTM) to obtain sequence features. Finally, the semantic information is transcribed through the connection-time classification (CTC) mechanism.

[0012] (6) Based on the semantic information obtained by the recognition module, perform arithmetic logic operations to determine whether each question is correct or not, and give the correct answer for questions that are answered incorrectly.

[0013] Preferably, step (1) first generates bounding boxes for the images using LabelImg annotation software, and then expands the original dataset using data augmentation techniques such as translation, rotation, and cropping to enhance the generalization and robustness of the data. For the bounding box information in the dataset, a total of four prior boxes at two scales are obtained using the K-means clustering algorithm, which are then used to train the detection network.

[0014] Preferably, step (2) uses the MixNet network to extract multi-scale features of the image. By replacing the depthwise separable convolutional structure with a hybrid depthwise convolutional structure of different kernels, the number of network parameters is greatly reduced while enhancing the network's ability to fuse multi-scale semantic and localization features, thereby extracting more systematic and comprehensive features. The MixNet network consists of hybrid depthwise separable convolutional modules (MDConv), which aim to fuse different convolutional kernel sizes into a single convolutional operation, making it easy to obtain different modes with multiple resolutions. Among them, the MDConv convolutional operation has several design options:

[0015] (2.1) Group size g: Determines the number of different types of convolutional kernels used for a single input tensor. In the extreme case of g=1, MDConv is equivalent to ordinary depthwise convolution. For MobileNets, MDConv can improve the accuracy and efficiency of the model when g=4.

[0016] (2.2) Kernel Size for Each Group: Theoretically, each group can have kernels of any size. However, if two groups have kernels of the same size, it is equivalent to merging the two groups into a single group. Therefore, each group must be restricted to have kernels of different sizes. Furthermore, since smaller kernels typically have fewer parameters and FLOPS, the kernel size is usually restricted to start at 3x3 and monotonically increases by 2 for each group. In other words, the kernel size for group i is typically 2i+1. For example, a 4-group MDConv typically uses kernel sizes of {3x3, 5x5, 7x7, 9x9}. Under this constraint, the kernel size for each group is predefined relative to its group size g, thus simplifying the design process.

[0017] (2.3) Channel size for each group: Two channel division methods are mainly considered here: (a) Equal division: the number of channels in each group is the same; (b) Exponential division: the number of channels in the i-th group accounts for a certain percentage of the total number of channels. For example, a 4-group MDConv has a total of 32 channels. The channels can be divided equally into (8,8,8,8), while the channels can be divided exponentially into (16,8,4,4).

[0018] Preferably, step (3) fuses features at different scales using Feature Pyramid Networks (FPN), and organically combines localization and semantic information through upsampling and channel concatenation, outputting feature maps with 8x and 16x downsampling, which are then fed into the YOLOv3 prediction module. The prediction module will determine the relative positions of the bounding boxes. Convert to absolute position using the following formula This facilitates comparison of the intersection-union ratio (IUU) of predicted and prior boxes, allowing predictions to be made based on the best prior box.

[0019]

[0020] The number of input feature map channels for the prediction module is (B x (5+C)), where B represents the number of bounding boxes that each unit can predict (here, we take 2). Each bounding box has 5+C attributes, describing its coordinate dimensions, confidence score, and class C probability. If the center of an object is located within the receptive field of a cell (the receptive field is the area of ​​the input image visible to the cell), then that cell is responsible for predicting the object. The loss function of the prediction module mainly consists of coordinate loss, confidence loss, and classification loss, as shown in the following formula:

[0021]

[0022] Where S^2 represents the number of grids at each scale, and B represents the number of bounding boxes predicted at each scale (taken as 2). This indicates whether the j-th box in the i-th grid is responsible for detecting this object. The first two terms represent the coordinate loss, using the mean squared error function. and The weights used for control center regression and width / height regression are generally set to 1 and 2; the third and fourth terms represent the confidence loss, using the cross-entropy function. Since the bounding boxes not responsible for detection have a higher weight, the weighting is adjusted by setting... =2 to accelerate the convergence of confidence; the last term represents the class loss, which uses the cross-entropy function, and each grid responsible for detection calculates the class probability.

[0023] Preferably, step (4) extracts each arithmetic problem based on the bounding box coordinates predicted by the detection module, and uses the labeled text information as a tag for training the recognition module.

[0024] Preferably, step (5) uses a CRNN model to extract semantic information from arithmetic problems. CRNN models are commonly used for end-to-end variable-length text sequence recognition. Instead of segmenting individual characters first, they transform text recognition into a time-dependent sequence learning problem. The workflow is as follows: given a single-channel grayscale image as input, it first passes through a convolutional neural network (CNN) to extract features, then converts it into a sequence and feeds it into a bidirectional long short-term memory network (LSTM) to obtain sequence features. Finally, it passes through a connected temporal classification (CTC) to transcribe and obtain the final label sequence. The last two pooling layers in the CNN are changed from 2x2 to 1x2. Considering that most of the text output by the detection module is relatively small in height and relatively long in width, using a 1x2 pooling window can ensure that information in the width direction is not lost as much as possible.

[0025] In the process of translating the sequence output by a Recurrent Neural Network (RNN) into the final result, a lot of redundant information will appear, such as a letter being recognized twice in a row. In this case, the blanking mechanism is needed to solve the redundant information in the RNN network output. By inserting a "-" (representing blank) between repeated characters, the same characters are merged (except for those separated by blanking characters), thus solving the problem of repeated characters.

[0026] Given the input probability distribution matrix of the RNN T is the sequence length, which is ultimately mapped to the label text. The probability is:

[0027]

[0028] in The transformation of the mapping function B from sequence to sequence is: The set of all paths is represented by π, where π is one of those paths. The probability of each path is the product of the probability distributions of the corresponding characters at each time step. The network is trained to maximize this probability value, and the loss function is defined as the negative maximum likelihood function of the probability. During the testing phase, the character with the highest probability at each time step is concatenated, and the final prediction result is obtained by applying the blanking mechanism described above.

[0029] Preferably, step (6) uses arithmetic logic operations to determine the correctness of each arithmetic problem based on the semantic information predicted by the recognition module, and provides the correct answer for the incorrectly answered problem.

[0030] Beneficial effects: The arithmetic problem grading system based on MixNet-YOLOv3 and convolutional recurrent neural network CRNN proposed in this invention can automatically identify the meaning of each arithmetic problem in the test paper and make a judgment in a very short time, reducing the manpower and time costs brought about by traditional manual grading of test papers and improving teaching efficiency. Attached Figure Description

[0031] Figure 1 This is a flowchart of an arithmetic problem grading system based on MixNet-YOLOv3 and Convolutional Recurrent Neural Network (CRNN).

[0032] Figure 2 This is a network architecture diagram of MixNet-YOLOv3.

[0033] Figure 3 This is the network structure diagram of a convolutional recurrent neural network (CRNN). Detailed Implementation

[0034] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings:

[0035] like Figure 1 As shown, this invention discloses an arithmetic problem grading system based on MixNet-YOLOv3 and a convolutional recurrent neural network (CRNN). The system mainly consists of two modules: detection and recognition. The detection module uses a MixNet-YOLOv3 network that fuses multi-scale semantic and localization features to extract the bounding box information of the arithmetic problems. The extracted arithmetic problems are then sent to the recognition module. The recognition module uses a convolutional recurrent neural network (CRNN) based on a connection-time classification (CTC) decoding mechanism to obtain the semantic information of the problems. Finally, the correctness of each problem can be determined through arithmetic logic. The invention is further described below with reference to specific implementations, mainly including the following steps:

[0036] Step A involves preprocessing the original detection dataset. First, bounding boxes are generated for the images using LabelImg annotation software. Then, data augmentation techniques such as translation, rotation, and cropping are used to expand the original dataset, enhancing its generalization and robustness. For the bounding box information in the dataset, K-means clustering analysis is used to obtain four prior boxes across two scales, which are then used to train the detection network.

[0037] Step B involves inputting the entire image into the MixNet network model for image localization and semantic feature extraction, resulting in multi-scale features representing global image information. The MixNet network replaces depthwise separable convolutional structures with mixed depthwise convolutional structures using different kernels, significantly reducing the number of network parameters while enhancing the network's ability to fuse multi-scale semantic and localization features, thereby extracting more systematic and comprehensive features. For example... Figure 2As shown, the MixNet network consists of Hybrid Depth-Separable Convolutional Modules (MDConv), designed to fuse different convolutional kernel sizes into a single convolutional operation, making it easy to capture different patterns with multiple resolutions.

[0038] Step C utilizes the Feature Pyramid Network (FPN) feature fusion technology to fuse features at different scales, which are then fed into the corresponding YOLOv3 prediction modules. The generated feature maps are calculated based on two sets of prior bounding boxes. Prediction is performed based on the best prior bounding box, and a loss function is calculated using the predicted coordinates, confidence level, category information, and label information. Iterative training yields a more accurate detection model. The Feature Pyramid Network (FPN) fuses features at different scales, using upsampling and channel concatenation to organically combine localization and semantic information, outputting 8x and 16x downsampling feature maps, which are then fed into the YOLOv3 prediction module. The prediction module will determine the relative positions of the bounding boxes. Convert to absolute position using the following formula This facilitates comparison of the intersection-union ratio (IUU) of predicted and prior boxes, allowing predictions to be made based on the best prior box.

[0039]

[0040] The number of input feature map channels for the prediction module is (B x (5+C)), where B represents the number of bounding boxes that each unit can predict (here, we take 2). Each bounding box has 5+C attributes, describing its coordinate dimensions, confidence score, and class C probability. If the center of an object is located within the receptive field of a cell (the receptive field is the area of ​​the input image visible to the cell), then that cell is responsible for predicting the object. The loss function of the prediction module mainly consists of coordinate loss, confidence loss, and classification loss, as shown in the following formula:

[0041]

[0042] in, B represents the number of grids at each scale, and B represents the number of bounding boxes predicted at each scale (taken as 2). This indicates whether the j-th box in the i-th grid is responsible for detecting the object. The first two terms represent the coordinate loss, using the mean squared error function. and The weights used for control center regression and width / height regression are generally set to 1 and 2; the third and fourth terms represent the confidence loss, using the cross-entropy function. Since the bounding boxes not responsible for detection have a higher weight, the weighting is adjusted by setting... =2 to accelerate the convergence of confidence; the last term represents the class loss, which uses the cross-entropy function, and each grid responsible for detection calculates the class probability.

[0043] Step D: Based on the arithmetic problem bounding boxes output by the detection network, the extracted arithmetic problems and labeled text information are combined to form the dataset for the recognition module. Then, the extracted arithmetic problem images are converted to grayscale and fed into a convolutional recurrent neural network (CRNN). First, features are extracted using a CNN to obtain feature maps, then converted into sequences and fed into a bidirectional long short-term memory (LSTM) network to obtain sequence features. Finally, the semantic information is obtained through connection-linked temporal classification (CTC). The last two pooling layers in the CNN are changed from 2x2 to 1x2. Considering that the text output by the detection module is mostly small in height and long in width, using a 1x2 pooling window can minimize the loss of information in the width direction.

[0044] Translating the sequence output by a Recurrent Neural Network (RNN) into the final result can result in a lot of redundant information, such as a letter being recognized twice. To address this redundancy, a blanking mechanism is needed. This is achieved by inserting a "-" (representing a blank) between repeated characters and merging identical characters (except those separated by blank characters). Given an input probability distribution matrix for the RNN... T is the sequence length, which is ultimately mapped to the label text. The probability is:

[0045]

[0046] in The transformation of the mapping function B from sequence to sequence is: The set of all paths is represented by π, where π is one of those paths. The probability of each path is the product of the probability distributions of the corresponding characters at each time step. Training maximizes this probability value, and the loss function is defined as the negative maximum likelihood function of the probability. During the testing phase, the character with the highest probability at each time step is concatenated, and the final prediction result is obtained using the blanking mechanism described above.

[0047] Step F involves performing arithmetic logic operations based on the semantic information obtained from the recognition module to determine the correctness of each question and provide the correct answer for incorrect questions.

[0048] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. An arithmetic question intelligent marking method based on MixNet-YOLOv3 and a convolutional recurrent neural network (CRNN), characterized in that, Comprise the following steps: (1) The original detection data set is preprocessed, the generalization of the sample is enhanced by data enhancement means, and two groups of prior boxes suitable for the scene are generated by using K-means clustering algorithm, which are used for training of the detection network; (2) The whole image is input into the lightweight MixNet network model, the image positioning and semantic feature extraction are carried out, and the multi-scale features representing the global information of the image are obtained; (3) The feature pyramid network FPN (Feature Pyramid Networks) feature fusion technology is used to fuse features of different scales, which are respectively sent to the corresponding YOLOv3 prediction module, and the generated feature maps are calculated according to the two groups of prior boxes, the best prior box is used for prediction, and the loss function is calculated according to the predicted coordinates, confidence, class information and label information, and the detection model is obtained through iterative training; (4) According to the arithmetic problem frame output by the detection network, the extracted arithmetic problem and the labeled text information are combined to form the data set of the recognition module; (5) The extracted arithmetic problem image is converted into a gray image and sent to the convolutional recurrent neural network CRNN network, first the convolutional neural network CNN (Convolutional Neural Networks) is used to extract features to obtain a feature map, then the sequence is converted and sent to the bidirectional long short-term memory network LSTM (Long Short-Term Memory) to obtain sequence features, and finally the connection time classification CTC (Connectionist Temporal Classification) algorithm is used to transcribe the final semantic information; (6) According to the obtained semantic information of the recognition module, the arithmetic logic operation is carried out, so as to judge whether each arithmetic problem is correct or not, and the correct answer is given for the wrong problem. Wherein, the step (3) fuses features of different scales through a feature pyramid network (FPN), combines the positioning information and the semantic information in an organic way by using up-sampling and channel splicing, outputs 8 times and 16 times down-sampled feature maps, and sends the feature maps into a prediction module of the YOLOv3. The relative position of the frame is converted into an absolute position by the following formula , so as to compare the intersection over union of the predicted frame and the prior frame and predict based on the best prior frame. The input feature map channel number of the prediction module is Wherein B represents the number of bounding boxes that can be predicted by each unit, each bounding box has 5+C attributes, respectively describing the coordinate size, confidence and Class probability, if the center of the object is located in the receptive field of the unit cell, the object is predicted by the unit cell, and the loss function of the prediction module is mainly composed of coordinate loss, confidence loss and classification loss, the formula is as follows: in, This represents the number of grid cells at each scale. This represents the number of bounding boxes predicted at each scale. Indicates the first The first grid Whether a box is responsible for detecting this object, the first two terms represent the coordinate loss, using the mean squared error function. and The weights used for control center regression and width / height regression are set to 1 and 2, respectively; the third and fourth terms represent the confidence loss, using the cross-entropy function. Since the bounding boxes not responsible for detection have a higher weight, the weighting is adjusted accordingly. This accelerates the convergence of confidence; the last term represents the class loss, which uses the cross-entropy function, and each grid cell responsible for detection calculates the class probability.

2. The arithmetic question marking method based on MixNet-YOLOv3 and convolutional recurrent neural network (CRNN) according to claim 1, characterized in that, The step (1) first generates the label frame of the picture by using the LabelImg labeling software, then uses the translation, rotation and cutting data enhancement means to expand the original data set, enhances the generalization and robustness of the data, and obtains 2 prior boxes with 4 scales through K-means clustering algorithm according to the frame information of the data set, which is used for training of the detection network.

3. The arithmetic question marking method based on MixNet-YOLOv3 and convolutional recurrent neural network (CRNN) according to claim 1, characterized in that, The step (2) uses the lightweight MixNet network to extract image multi-scale features, replaces the depth separable convolution structure with the mixed depth convolution structure of different convolution kernels, reduces the network parameter amount, enhances the ability of the network to fuse multi-scale semantic and positioning features, so as to extract more systematic and comprehensive features, and the mixed depth separable convolution module MDConv (Mixed Depthwise Convolutional Kernel) aims to fuse different convolution kernel sizes into a single convolution operation, which is easy to obtain different modes with multiple resolutions, wherein the MDConv convolution operation has multiple design options: The size of the (2.1) group : determines the number of different types of convolution kernels for a single input tensor, in the extreme case, MDConv is equivalent to a normal deep convolution, in MDConv can improve the accuracy and efficiency of the model; (2.2) Convolution kernel size per group: limit the convolution kernel size from Initially, each group is monotonically increased by 2, i.e. The convolution kernel size of the group is ; (2.3) Channel size for each group: Two channel division methods are used: (a) Equal division: the number of channels in each group is the same; (b) Exponential division: the number of channels in each group is the same. The number of group channels accounts for the total number of channels. .

4. The arithmetic question marking method based on MixNet-YOLOv3 and convolutional recurrent neural network (CRNN) according to claim 1, characterized in that, The step (4) extracts each arithmetic question according to the bounding box coordinates predicted by the detection module, and labels the text information as a label for training of the recognition module.

5. The arithmetic question marking method based on MixNet-YOLOv3 and convolutional recurrent neural network (CRNN) according to claim 1, characterized in that, The step (5) extracts semantic information of the arithmetic question by using a convolutional recurrent neural network (CRNN) model. The CRNN model is commonly used for end-to-end recognition of indefinite-length text sequences. The text recognition is converted into a sequence learning problem with time dependence without first segmenting individual characters. The workflow is as follows: given a single-channel grayscale image as input, features are extracted by a convolutional neural network (CNN) to obtain a feature map, the sequence is converted and fed into a bidirectional long short-term memory (LSTM) network to obtain sequence features, and finally, a connectionist temporal classification (CTC) mechanism is used to transcribe the final label sequence. The last two pooling windows of the CNN are 1x2. In the process of translating the sequence output by a recurrent neural network (RNN) into a final recognition result, a lot of redundant information may occur, i.e., a letter is recognized twice in succession, and at this time, a blank mechanism needs to be used to solve the redundant information existing in the RNN output, by inserting a "-" between the repeated characters, the same characters are merged, and thus the problem of repeated characters is solved. For the probability distribution matrix given by the RNN , T is the sequence length, and the probability of finally mapping into a label text is: in The transformation of the mapping function B from sequence to sequence is: The set of all paths, and This is one of the paths. The probability of each path is the product of the probability distribution of the corresponding character at each time step. The network is trained to maximize this probability value, and the loss function is defined as the negative maximum likelihood function of the probability. In the testing phase, the character with the highest probability at each time step is concatenated, and the final prediction result can be obtained according to the blank mechanism mentioned above.

6. The arithmetic question marking method based on MixNet-YOLOv3 and convolutional recurrent neural network (CRNN) according to claim 1, characterized in that, The step (6) determines whether each arithmetic question is correct or not by using arithmetic logic operations based on the semantic information predicted by the recognition module, and provides the correct answer for the wrong questions.

Images