Handwritten chinese character recognition method based on fusion-based capsule network and light-weight network

Abstract

The application discloses a handwriting Chinese character recognition method based on a fusion capsule network and a lightweight network, and comprises the following steps: data preprocessing; building and training an Efficient-Capsnet capsule network model; building and training a lightweight MobileNetV3 network model; building an EfficientNetV2 model, and performing stacking model training; using a stacking model fusion mode, taking the Efficient-Capsnet capsule network model and the lightweight MobileNetV3 network model as primary learners, and inputting the outputs of the primary learners into a secondary learner, i.e., the EfficientNetV2 model, to perform classification and finally recognize handwriting Chinese characters. The application adopts a method of fusing a capsule network and a lightweight MobileNetV3 to recognize handwriting Chinese characters, so that the model parameter quantity is reduced, the spatial recognition capability is increased, and the accuracy of the handwriting Chinese character recognition model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of OCR, and particularly relates to a handwriting Chinese character recognition method based on a fusion capsule network and a lightweight network. BACKGROUND

[0002] In recent years, handwriting Chinese character recognition has gradually emerged. Due to the particularity of Chinese characters, such as a large number of types, various fonts, and various writing methods, handwriting Chinese character recognition is a very difficult problem for machine learning. After years of efforts by many researchers, the handwriting Chinese character recognition field has made a qualitative leap, and the recognition accuracy of single Chinese characters and single-line Chinese characters has been greatly improved.

[0003] However, in recent years, due to the lack of development space for template matching and feature value extraction, handwriting Chinese character recognition has obtained a new solution of deep learning. With the development of deep learning, handwriting Chinese character recognition has a great improvement space, but also brings some problems, such as: too large image, too deep network layer, too large parameter quantity, and too long training time. A large number of experiments have proved that under the support of deep learning technology, the recognition rate of handwriting Chinese characters is significantly improved. In such an environment, researchers in the field of image recognition use various methods to improve the accuracy of handwriting recognition. In 2011, the handwriting Chinese character recognition competition of the 11th International Conference on Document Analysis and Recognition held by Tsinghua University and the Institute of Automation of the Chinese Academy of Sciences, the highest recognition rate of offline handwriting Chinese character recognition was 93.01%, and in 2013, the team of Fujiwon obtained a recognition rate of 93.58% using the method of convolutional neural network.

[0004] Because of the method of deep learning, the recognition rate of handwriting Chinese characters has been significantly improved in recent years, but it also brings some problems, such as: too large image, too many network layers, and too large parameter quantity. In addition, handwriting Chinese characters are written randomly and are not standardized, Chinese characters are complex and have a lot of variations. Many Chinese characters are highly similar in appearance and are easily confused. The written Chinese characters are stacked, making it difficult for machines to normally recognize and separate characters. The characters at the light-dark junction are difficult to recognize; the handwriting Chinese character recognition problem of super large class: the character types of Chinese characters are very large, and the data required for training is very large; Chinese characters have a long history, and brushstroke, shorthand, and other effects will make recognition more difficult; it is still very difficult to correctly recognize under complex and diverse interference. SUMMARY

[0005] In order to solve the problems in the prior art, the present application aims to provide a handwriting Chinese character recognition method based on a fusion capsule network and a lightweight network, which reduces the model parameter quantity while increasing the spatial recognition capability, and can also improve the accuracy of the handwriting Chinese character recognition model.

[0006] The present application solves the above problems by the following technical solutions:

[0007] The handwriting Chinese character recognition method based on the fusion capsule network and the lightweight network comprises the following steps:

[0008] Step A, data preprocessing;

[0009] The data of the Chinese handwriting database is acquired, and the acquired data is combined and used to expand the data set, the data set is divided into a training data set and a test data set in proportion, and the data sample size is unified;

[0010] Step B, building and training an Efficient-Capsnet capsule network model;

[0011] Step C, building and training a lightweight MobileNetV3 network model;

[0012] Step D, building an EfficientNetV2 model, and training a stacking fusion model;

[0013] The stacking model fusion method is used, the Efficient-Capsnet capsule network model and the lightweight MobileNetV3 network model are used as primary learners, and the EfficientNetV2 model is used as a secondary learner;

[0014] The test data set after preprocessing is input into the trained Efficient-Capsnet capsule network model and the lightweight MobileNetV3 network model for testing, and the prediction output of the test data set is obtained;

[0015] The prediction output is input into the trained secondary learner EfficientNetV2 model for classification to complete the final handwriting Chinese character recognition.

[0016] As a further improvement of the present application, in step A, it comprises:

[0017] The data set is labeled by using One-Hot one-hot encoding, and the paths and labels of the training data set and the test data set are stored in train.txt and test.txt texts, so as to facilitate the input of the labels of the data set into the model.

[0018] As a further improvement of the application, the Efficient-Capsnet capsule network model is built, and the specific method comprises:

[0019] The conv convolution layer uses 256 9x9x1 convolution kernels with a stride of 1, and uses a ReLU activation function; the Primary main capsule layer combines the basic features detected by the conv convolution layer; the Self-Attention self-attention mechanism generates a coupling coefficient matrix by using a self-attention algorithm, and the coupling coefficient is calculated from a self-attention tensor, thereby forming a non-iterative routing operation based on the self-attention mechanism; the Digital digital capsule layer; three fully connected layers constitute the entire Efficient-Capsnet capsule network; the input vector is subjected to matrix multiplication, the input vector is subjected to scalar weighting, and the weighted vector is summed, and a Squash activation function is used to realize the non-linearization of the vector to the vector, so as to build the Efficient-Capsnet capsule network model.

[0020] As a further improvement of the application, the lightweight MobileNetV3 network model is built, and the specific method comprises:

[0021] The MobileNetV3 network is used, the depth separable convolution is used instead of the traditional convolution, the convolution with stride=2 is used to replace the pooling, the inverted residual structure and the linear bottleneck are introduced, and the SE channel attention structure is introduced, the Relu6(x+3) / 6 is used to approximate the sigmoid in the SE module, the NetAdapt algorithm is used to obtain the optimal number of convolution kernels and channels, the parameters are obtained by NAS search, and the activation function hard-swish(x) is used instead of Relu6, so as to build the lightweight MobileNetV3 network model.

[0022] As a further improvement of the application, the stacking model fusion method is used, the Efficient-Capsnet capsule network model and the lightweight MobileNetV3 network model are used as primary learners, and the EfficientNetV2 model is used as a secondary learner, and the specific method comprises:

[0023] The outputs of the fully connected layers of the Efficient-Capsnet capsule network model and the lightweight MobileNetV3 network model are input into the EfficientNetV2 model, and the Conv2D convolution layer, the Fuse-MBConv convolution layer, the MBConv convolution layer and the fully connected layer structure of the EfficientNetV2 model are used for training of the secondary learner, so as to complete the stacking fusion model training.

[0024] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0025] The present application adopts the method of fusing capsule network and lightweight MoblieNetV3 to perform handwritten Chinese character recognition, reduces the model parameter quantity, and also increases the spatial recognition ability to improve the accuracy of the handwritten Chinese character recognition model. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 The present application is a handwritten Chinese character image data set situation display;

[0027] Figure 2 The present application is a handwritten Chinese character recognition method based on the fusion of capsule network and lightweight model flowchart;

[0028] Figure 3 The present application is a simple structure diagram of the self-attention mechanism of the Efficient-Capsnet capsule network structure;

[0029] Figure 4 The present application is a structure diagram of the basic structure module of the MobileNetV3 network, the Efficient-Capsnet network, the EfficientNetV2 model and the stacking model fusion process. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0031] Embodiment 1:

[0032] Referring to the accompanying drawings Figures 1-4 A handwritten Chinese character recognition method based on the fusion of capsule network and lightweight network, the specific implementation environment is: Anaconda3, python 3.7, pycharm (IDE), pytorch GPU. The specific implementation method includes the following steps:

[0033] Step 1: data preprocessing;

[0034] Obtain the data of the Chinese handwritten database, and combine the obtained data to expand the data set, divide the data set into training data set and test data set according to the proportion, and unify the data sample size;

[0035] Specifically, the CASIA offline Chinese handwriting database of the Institute of the Chinese Academy of Sciences is used to download the HWDB1.1 dataset. Among them, the commonly used 3754 Chinese character categories are in the HWDB1.1 dataset folder, the HWDB1.1 dataset contains 3754 class folders, and the One-Hot one-hot encoding is used to take 1-3754 as the label of the dataset, and each category contains about 300 image data; in this embodiment, the handwritten Chinese dataset SCUT-EPT Dataset dataset of South China University of Technology is also used, the HWDB1.1 dataset and the SCUT-EPT Dataset dataset are combined to combine the two datasets for data set expansion; then, the training dataset and the test dataset are divided in the ratio of 8:2, and the data is unified in size.

[0036] The paths and labels of the training dataset and the test dataset are stored in train.txt and test.txt texts to facilitate the label input of the dataset into the model, and each line of the text represents the path name and label of each category of Chinese characters as shown in Figure 1 , so that the labels and data of the dataset are displayed.

[0037] Step two: build and train the Efficient-Capsnet capsule network model;

[0038] By constructing an input layer, a Conv2D convolution layer, a DepthwiseConv2D layer, a Primary primary capsule network, and a self-attention layer with Self-Attention, the structure is used to effectively route the number and reduce the number of Capsnet as shown in Figure 3 The weight transfer structure diagram of the self-attention layer and the capsule layer, the Digital digital capsule layer, and the fully connected layer finally complete the building of the Efficient-Capsnet network model. Using the Efficient-Capsnet capsule network model can obtain spatial recognition ability by using the capsule network, and also can reduce the capsule network parameters by using the self-attention structure without affecting the accuracy of the model.

[0039] Among them, Figure 3 is a simple structure diagram of the self-attention mechanism structure of the Efficient-Capsnet capsule network structure, represents the number of capsules n l in the lth layer, d l represents the dimension of each capsule; represents the weight matrix, which includes the affine transformation of the two layers of capsules before and after; represents the logarithmic prior matrix; represents the coefficient matrix, which contains all the coupling coefficients generated by the self-attention algorithm; represents the mapping result of all the bottom capsules to the upper capsules, as shown in formula (1).

[0040]

[0041] The calculation formula of the l+1 layer capsule is as follows, wherein the prior matrix contains all the weight discrimination information. The self-attention algorithm generates a coupling coefficient matrix The coupling coefficient is calculated from the self-attention tensor, and the specific formula is shown in formula (2).

[0042]

[0043] Each capsule contains a matrix, is used to help stabilize training and balance coupling coefficients and priors. As shown in formula (3), it is used to calculate the final coefficients required by the capsule in the l+1 layer.

[0044]

[0045] The role of the coupling coefficient in the l layer and the l+1 layer is to obtain the routing weight, and the calculation formula is shown in formula (4).

[0046]

[0047] The output layer of the Efficient-Capsnet is no longer a scalar, but a vector. The squash activation function realizes the nonlinearization of the vector to the vector. The squash squeeze activation function is shown in formula (5). The function function is to make the length of v j not more than 1, and keep v j and s j in the same direction.

[0048]

[0049] The Efficient-Capsnet capsule network model is built, and the specific method comprises:

[0050] The primary capsule layer combines the basic features detected by the conv convolution layer; the self-attention mechanism generates a coupling coefficient matrix using a self-attention algorithm, and the coupling coefficient is calculated from the self-attention tensor, thereby forming a non-iterative routing operation based on the self-attention mechanism; the digital capsule layer; three fully connected layers constitute the entire Efficient-Capsnet capsule network. The input vector is multiplied by a matrix, the input vector is scalar weighted, and the weighted vector is summed, and the Squash activation function is used to realize the non-linearization of the vector to the vector, so as to build the Efficient-Capsnet capsule network model.

[0051] The built Efficient-Capsnet capsule network model is instantiated, the model is set to a training mode, the 3754-class Chinese character image training data set data preprocessed in step one is input into the efficient self-attention routing Efficient-Capsnet capsule network model, an optimizer class is instantiated, a loss function is instantiated, a traversal dataloader is obtained, the gradient is initialized to 0, and forward calculation is performed. The main capsule layer is obtained by depth separable convolution, which greatly reduces the number of parameters required to make capsules, and a non-iterative and highly parallel routing algorithm is used instead of dynamic routing, and the self-attention mechanism reduces the number of capsules. This algorithm can also use the self-attention mechanism to effectively route. The vector output by each Capsule not only represents the probability of the class, but also contains information of each attribute, and the objective function is shown in formula (6). The loss is calculated using the loss function, the loss value is back propagated, the model learning parameters are further updated, and the loss value gradually decreases and tends to converge during the iterative calculation process, thereby completing the training of the model.

[0052]

[0053] wherein, is a loss function, m + , m - , λ are hyperparameters, is the attribute information of the n capsules of the L layer.

[0054] Step three: build and train a lightweight MobileNetV3 network model;

[0055] The light MobileNetV3 network model is built by constructing an input layer, a Conv2D convolution layer, a bottleneck inverse residual structure layer with an attention mechanism SE (Squeeze-and-Excitation) structure, using a mean pooling layer and a final fully connected layer. The light MobileNetV3-small model is used to realize image recognition of a large data set, make up for the weakness of the capsule network in large data sets, and improve the recognition accuracy of handwritten Chinese characters.

[0056] The light MobileNetV3 network model is built, and the specific method comprises the following steps:

[0057] The MobileNetV3 network is used. In the network, the Efficient-Capsnet capsule network model uses depthwise separable convolution instead of traditional convolution, uses stride=2 convolution instead of pooling, uses an inverse residual structure and a linear bottleneck, and introduces an SE channel attention structure. Relu6(x+3) / 6 is used to approximate the sigmoid in the SE module. The NetAdapt algorithm is used to obtain the optimal number of convolution kernels and channels. The parameters are obtained by NAS search. The hard-swish(x) activation function is used to replace Relu6 to build the light MobileNetV3 network model.

[0058] The constructed light MobileNetV3 network model is instantiated. The light MobileNetV3 network model is set to a training mode. The preprocessed 3754-class Chinese character image training data set in step one is input into the input layer of the MobileNetV3 network for training. The constructed MobileNetV3 model is instantiated. The model is set to a training mode. The traversal dataloader is obtained. The gradient is initialized to 0. Forward calculation is performed. The depthwise separable convolution improves the calculation efficiency. The linear bottleneck extracts features from a high-dimensional space without losing too much information. The inverse residual is a safer and more effective method to extract all necessary information of the input data, which replaces the ReLU layer. The network architecture search is applied to the network through reinforcement learning and recurrent neural networks. The resource-constrained NAS performs module-level search. The NetAdapt performs local search. The network optimization is performed in an automatic manner. The resource consumption is gradually reduced while the accuracy is maximized. The average pooling layer in the last step is moved forward, and the last convolution layer is removed. The h-swish activation function is introduced to obtain higher accuracy. The h-swish(x) function is shown in equation (7). Similarly, the loss is calculated by using the loss function. The model learning parameters are further updated by the loss. The loss value gradually decreases and converges during the iterative calculation process, so as to complete the training of the model.

[0059]

[0060] Thus the training of the lightweight MobileNetV3 network model is completed.

[0061] Step four: build the EfficientNetV2 model and train the stacking fusion model;

[0062] 4.1, using the stacking model fusion method, the Efficient-Capsnet capsule network model and the lightweight MobileNetV3 network model are used as the primary learner, and the EfficientNetV2 model is used as the secondary learner.

[0063] The outputs of the full connection layers of the Efficient-Capsnet capsule network model and the lightweight MobileNetV3 network model are input into the EfficientNetV2 model, and the Conv2D convolution layer, Fuse-MBConv convolution layer, MBConv convolution layer and full connection layer structure of the EfficientNetV2 model are used for training the secondary learner. The unique progressive learning strategy makes the model training faster, and the training of the stacking fusion model is completed.

[0064] 4.2, the test data set after preprocessing is input into the Efficient-Capsnet capsule network model and the lightweight MobileNetV3 network model for prediction to obtain the prediction results of the test data set.

[0065] Specifically, the test data set is input into the stacking model for testing, and according to the prediction accuracy of the test data set on the model, the learning rate is dynamically adjusted using the ReduceLROnPlateau method, the AdamW optimizer is used for weight decay, which does not participate in the calculation of the first and second momentum, and is only used in the final update formula. Adjust the accuracy of the fusion model to more than 95%. The stacking uses multi-fold cross-validation, which is more robust than using a single model. The overall network architecture is as shown in Figure 4 .

[0066] 4.3, the prediction results are input into the secondary learner EfficientNetV2 model for classification to complete the final handwritten Chinese character recognition.

[0067] The image test data set is input into the integrated fusion model, the created network model is instantiated, the trained model weight parameter handwriting.pth is loaded, the weight parameter is imported into the network model, and the model evaluation mode is entered, after dimension compression, the last Softmax calculation is performed, the labels of the first five prediction results of the model output test data, the corresponding Chinese character types and prediction probabilities are set, and the console result of the example test is as shown in Figure 4 Of course, the result with the highest probability can be directly selected, and when the input is a text line, the prediction result with the highest probability of each character involved in the text is displayed on the console. In this way, the handwritten Chinese character recognition is completed.

[0068] The EfficientNetV2 model improves the training speed as much as possible while keeping the efficient use of the parameter amount, introduces Fused-MBConv into the search space on the basis of EfficientNet, and introduces an adaptive regular strength adjustment mechanism for progressive learning, so that the training speed is faster and the parameter amount is smaller.

[0069] The present application aims at the problem that a single capsule network is difficult to have good performance on a large data set, and the spatial recognition ability of a single deep learning network is usually poor, the information is lost by the pooling operation, and the parameter amount of the model running is large. The method of fusing the capsule network and the lightweight MoblieNetV3 is used for handwritten Chinese character recognition, the parameter amount of the model is reduced, the spatial recognition ability is increased, and the accuracy of the handwritten Chinese character recognition model is improved.

[0070] Although the present application is described herein with reference to the explanatory embodiments of the present application, the above-described embodiments are only preferred embodiments of the present application, and the embodiments of the present application are not limited by the above-described embodiments, and it should be understood that those skilled in the art can design many other modifications and embodiments, which will fall within the scope and spirit of the principles disclosed in the present application.

Claims

1. A method for handwritten Chinese character recognition based on a fusion capsule network and a lightweight network, characterized in that, Comprise the following steps: Step A, data preprocessing; Obtain the data of the Chinese handwritten database, and merge the obtained data set to expand the data set, divide the data set into training data set and test data set in proportion and unify the data sample size; Step B, build and train the Efficient-Capsnet capsule network model; Step C, build and train the lightweight MobileNetV3 network model; Step D, build the EfficientNetV2 model, and train the stacking fusion model; Use the stacking model fusion method, and use the Efficient-Capsnet capsule network model and the lightweight MobileNetV3 network model as the primary learner; the EfficientNetV2 model is used as the secondary learner; The test data set after preprocessing is input into the trained Efficient-Capsnet capsule network model and lightweight MobileNetV3 network model for testing, and the prediction output of the test data set is obtained; The prediction output is input into the trained secondary learner EfficientNetV2 model for classification to complete the final handwritten Chinese character recognition. 2.The method of claim 1, wherein, In step A, it comprises: Use One-Hot one-hot encoding to set the label of the data set, and store the path and label of the training data set and test data set in train.txt and test.txt text for easy input of the data set label into the model.

3. The method according to claim 1, wherein the method comprises: The Efficient-Capsnet capsule network model is built, and the specific method comprises: Use the conv convolution layer to use 256 9x9x1 convolution kernels and a step of 1, and use the ReLU activation function; the Primary main capsule layer combines the basic features detected by the conv convolution layer; the Self-Attention self-attention mechanism generates a coupling coefficient matrix by using the self-attention algorithm, and the coupling coefficient is calculated from the self-attention tensor, thereby forming a non-iterative routing operation based on the self-attention mechanism; Digital digital capsule layer; three fully connected layers constitute the entire Efficient-Capsnet capsule network; matrix multiplication is performed on the input vector, the input vector is weighted by a scalar, and the weighted vector is summed, and the Squash activation function is used to realize the non-linearization of the vector to the vector, so as to build the Efficient-Capsnet capsule network model.

4. The method according to claim 1, wherein the method comprises: The lightweight MobileNetV3 network model is built, and the specific method comprises: Using MobileNetV3 network, using depth separable convolution instead of traditional convolution, using stride=2 convolution instead of pooling, using reverse residual structure and linear bottleneck and introducing SE channel attention structure, using Relu6(x+3) / 6 to approximate sigmoid in SE module, using NetAdapt algorithm to obtain the optimal number of convolution kernels and channels, using the activation function hard-swish(x) instead of Relu6 to replace Relu6, and using the activation function hard-swish(x) instead of Relu6 to build a lightweight MobileNetV3 network model.

5. The method according to any one of claims 1-4, wherein the method comprises: The stacking model fusion method uses the Efficient-Capsnet capsule network model and the lightweight MobileNetV3 network model as the primary learner, and the EfficientNetV2 model as the secondary learner, and specifically comprises: The output of the full connection layer of the Efficient-Capsnet capsule network model and the lightweight MobileNetV3 network model is input into the EfficientNetV2 model, and the Conv2D convolution layer, the Fuse-MBConv convolution layer, the MBConv convolution layer and the full connection layer structure of the EfficientNetV2 model are used for training of the secondary learner, so as to complete the stacking fusion model training.

Images