A method for recognizing handwritten Chinese text

Abstract

The application discloses a kind of handwritten Chinese text recognition method, belong to image data processing field, including the following steps: obtaining handwritten Chinese dataset D1, wherein sample is handwritten text picture, and character in sample is randomly partially marked;A segmentation module is constructed;A handwritten Chinese text recognition network is constructed, including the segmentation module, recognition module, regular module connected in turn;Handwritten Chinese text recognition network is trained using D1, and handwritten Chinese text recognition model is obtained;Obtain the handwritten Chinese text to be identified, and the corresponding final text is obtained by handwritten Chinese text recognition model.The application constructs a kind of brand-new segmentation module, can also generate positioning result to unmarked character, achieve without manual marking real frame to generate recognition result, reduce the cost of model actual landing, avoid to mark a large number of data, reduce cost, improve work efficiency.And the application can ensure that semantic context is smooth and fluent, and the final text accuracy is high.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data processing, and in particular to a handwritten Chinese text recognition method. BACKGROUND

[0002] Handwritten Chinese character recognition is to process a text image to obtain the characters contained in the text image. Handwritten Chinese character recognition initially mainly studies single character recognition, which is divided into a traditional recognition method and a deep learning-based recognition method. The traditional single character recognition method mainly includes three steps of preprocessing, feature extraction, and recognition classification. The deep learning-based single character recognition mainly uses a convolutional neural network, and has currently achieved a high recognition accuracy. Subsequently, handwritten Chinese character recognition based on text lines has gradually developed. The number of characters in a text line image is indefinite and is prone to phenomena such as character slant, excessively far or close character spacing, and the like, which increase the difficulty of handwritten Chinese character recognition based on text lines. At present, there are mainly two types of recognition methods for handwritten Chinese character recognition based on text lines.

[0003] One type is a character segmentation-based recognition method, which first divides a text line picture into a plurality of small blocks, then performs recognition processing on each small block, and finally connects the recognized characters. First, a connected component-based method is used to over-segment a text line picture into a series of original segments, then the original segments are recognized to obtain a candidate character set, and a path is evaluated from a Bayesian decision view by combining multiple contexts (including character classification scores, geometry, and language context) to output a recognition result. This method improves the accuracy and efficiency through an improved beam search algorithm, and the character set accuracy reaches 90.75% on the CASIA-HWDB dataset. However, this method (1) relies on a sliding window or a stroke merging algorithm (such as HMM, stroke bounding box merging), has high computational complexity and poor robustness to connected characters, and (2) requires character-by-character annotation of training data, which has high annotation cost, low annotation efficiency, and is easily affected by segmentation errors to affect recognition accuracy.

[0004] Another type is a segmentation-free text recognition method, such as a segmentation-free scene text recognition method based on CTC (referred to as a CTC recognition algorithm) and a segmentation-free scene text recognition method based on an attention mechanism (referred to as an attention mechanism recognition algorithm). This type of method does not require character segmentation annotation, but is based on a segmentation strategy. It first uses a bidirectional long short-term memory network to extract features, then uses a dynamic programming algorithm to find the optimal character segmentation point, and finally uses a convolutional neural network to recognize each character. This method can avoid annotation dependence, but the CTC recognition algorithm ignores the spatial relationship between characters, making it difficult to simulate human character-by-character reading cognitive logic, and the attention mechanism recognition algorithm has a quadratic growth in computational overhead with sequence length, making it difficult to meet real-time requirements and limiting inference speed.

[0005] Noun explanation:

[0006] ResNet-18: ResNet (Residual Network) is a deep neural network structure that introduces residual connections to solve the problem of gradient vanishing and representation bottleneck in the training process of deep networks. ResNet-18 is a lightweight model in the ResNet series, which contains 18 convolutional layers and 1 fully connected layer.

[0007] BERT model: BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained natural language processing model. The core of the BERT model is the Transformer encoder, which can be pre-trained on a large corpus of text in an unsupervised manner, and then fine-tuned on various NLP tasks. BERT model is a bidirectional deep learning model that can consider all words in the context simultaneously, thus better understanding the meaning of the sentence. The pre-training tasks of BERT model include MLM task and NSP task. MLM (Masked Language Model) is also known as Masked Language Modeling, in which BERT randomly masks about 15% of the words and uses a bidirectional Transformer architecture to predict these masked words. The purpose is to force the encoder to use both left and right information to learn deep bidirectional representations. NSP (Next Sentence Prediction) is also known as Next Sentence Prediction, which is used to determine whether sentence B is the real next sentence following sentence A. The goal is to make the [CLS] vector have the ability to model sentence-level semantics and inter-sentence relationships, directly serving downstream tasks such as question answering and reasoning.

[0008] FCN (Fully Convolutional Network) is a class of neural networks whose all layers are composed of convolutional layers (as well as nonlinear, normalization, upsampling, etc. operations). Compared with other CNN convolutional neural network methods, its advantage is that it has no requirements for the size of the input data, and does not need to consider the size and pixel size of the input image. Summary of the invention

[0009] The purpose of the present application is to provide a handwritten Chinese text recognition method that can greatly reduce the labeling cost, improve the recognition efficiency and accuracy.

[0010] In order to achieve the above purpose, the technical solution adopted by the present application is as follows: a handwritten Chinese text recognition method, comprising the following steps:

[0011] S1, obtaining a handwritten Chinese dataset D1, wherein samples are handwritten text pictures, and characters in the samples are randomly partially labeled;

[0012] S2, constructing a segmentation module, including S21-S24;

[0013] S21, obtaining a target detection network M1 including a pre-trained ResNet-18 and FCN, for extracting image features of the handwritten text through the ResNet-18, and predicting the positioning results of each character in the handwritten text through the FCN, wherein the positioning results include predicted boxes and corresponding confidence levels;

[0014] S22, presetting an iteration number T, a first threshold θ1, a second threshold θ2, 0<θ1<θ2<1, and a set G;

[0015] S23, training M1, wherein the t-th training includes Sa1-Sa3, 1≤t≤T;

[0016] Sa1, inputting the sample X into M1 to obtain the positioning results of each character, and calculating the positioning loss L loc to minimize L loc adjusting the network parameters of M1;

[0017] Sa2, generating the first region R 1,t , the second region R 2,t , and the third region R 3,t on the sample X for the t-th training, specifically, the range of the predicted box with a confidence level less than θ1 constitutes R 1,t , the range of the predicted box with a confidence level greater than θ2 constitutes R 3,t , and the range of the remaining predicted box constitutes R 2,t ;

[0018] Sa3, if t=1, completing the training, and if t>1, sequentially updating the predicted box and the confidence level in R 2,t , updating a predicted box B 2,t in R t , and if the updated confidence level is greater than θ2, updating B t to R 3,t ;

[0019] S24, when t=T, ending the iteration, clipping the predicted box in R 3,t to obtain a single word picture, storing it in the set G, and storing the trained target detection network M1 as the segmentation module;

[0020] S3, constructing a handwritten Chinese text recognition network including a segmentation module, a recognition module, and a regularization module connected in sequence;

[0021] The recognition module adopts an FCN, which is used for inputting a single character graph in the set G, outputting a predicted character category, and arranging the coordinates of the positioning frame into an initial text;

[0022] The regular module is a BERT model pre-trained through an MLM task and an NSP task, which is used for correcting the initial text to generate a final text;

[0023] S4, training the handwritten Chinese text recognition network with D1, calculating the positioning loss of the segmentation module, the classification loss of the recognition module and the semantic loss of the regular module each time, and adjusting the handwritten Chinese text recognition network by minimizing the total loss to obtain a handwritten Chinese text recognition model;

[0024] S5, obtaining a handwritten Chinese text to be recognized, and obtaining a corresponding final text through the handwritten Chinese text recognition model.

[0025] Preferably, the handwritten Chinese data set comprises an offline handwritten Chinese data set CASIA-HWDB, an online handwritten Chinese data set CASIA-OLHWDB, an ICDAR2013 competition test set, a complex scene handwritten Chinese data set SCUT-HCCDoc and a scene Chinese data set ReCTS.

[0026] Preferably, θ1 is 0.4-0.5, and θ2 is 0.8-0.9.

[0027] Preferably, in Sa3, R 2,t a prediction frame B t , and the updating method is:

[0028] If B t has an overlap with the generated prediction frame B t-1 of the t-1th time, the confidence p t of B t and the weighted average value of the confidence p t-1 of B t-1 are calculated. B is updated; t

[0029] Comparing p t-1 and p , if p >p t-1 , B t is unchanged, otherwise, B t-1 is updated by B t .

[0030] Compared with the prior art, the present application has the following advantages:

[0031] ​(1) The application constructs a brand-new segmentation module, which needs to be trained twice. The first time is the pre-training of the target detection network M1, which does not need a large amount of data training, but only needs to use a small part of the labeled data for pre-training, and has a certain character segmentation capability. The second time is to train the target detection network M1 using the data set D1 according to the training method described in step S23 of the application. The prediction frame and the confidence of the character are constantly updated, and the final positioning result is generated. The positioning result can generate positioning results for unlabeled characters, can generate recognition results without manual labeling of real frames, reduces the actual landing cost of the model, avoids labeling a large amount of data, reduces the cost, and improves the work efficiency.

[0032] (2) After generating the positioning result of the character in the sample, the recognition module outputs the predicted character category, and then sends it into the regular module to correct the single word with low recognition probability, so as to ensure that the semantic context is smooth and fluent, and the final text accuracy is high. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The flowchart of the application;

[0034] Figure 2 The handwritten Chinese text recognition network structure diagram;

[0035] Figure 3 The handwritten text picture in Example 2 is recognized by step S23 process diagram;

[0036] Figure 4 The final text obtained by the application from picture 1;

[0037] Figure 5 The final text obtained by the application from picture 2;

[0038] Figure 6 The final text obtained by the application from picture 3;

[0039] Figure 7 The final text obtained by the application from picture 4. DETAILED DESCRIPTION

[0040] The application will be further described below in combination with examples and drawings.

[0041] Example 1: see Figure 1 and Figure 2 A handwritten Chinese text recognition method, comprising the following steps:

[0042] S1, obtaining a handwritten Chinese data set D1, wherein the sample is a handwritten text picture, and the characters in the sample are randomly labeled;

[0043] S2, constructing a segmentation module, comprising S21-S24;

[0044] S21, obtaining a target detection network M1 comprising a pre-trained ResNet-18 and FCN, for extracting image features of handwritten text through ResNet-18, and predicting the positioning results of each character in the handwritten text through FCN, wherein the positioning results comprise a prediction box and a corresponding confidence;

[0045] S22, presetting an iteration number T, a first threshold θ1, a second threshold θ2, 0<θ1<θ2<1, and a preset set G;

[0046] S23, training M1, wherein the t-th training comprises Sa1-Sa3, 1≤t≤T;

[0047] Sa1, inputting the sample X into M1 to obtain the positioning results of each character, and calculating the positioning loss L loc to minimize L loc adjusting the network parameters of M1;

[0048] Sa2, generating the first region R 1,t , the second region R 2,t , and the third region R 3,t of the t-th training on the sample X, specifically, the prediction box range with a confidence less than θ1 constitutes R 1,t , the prediction box range with a confidence greater than θ2 constitutes R 3,t , and the remaining prediction box range constitutes R 2,t ;

[0049] Sa3, if t=1, completing the training, and if t>1, updating R 2,t in turn, updating the prediction box and the confidence in R 2,t , and updating a prediction box B t in R t , if the updated confidence is greater than θ2, updating B 3,t to R 3,t ;

[0050] S24, when t=T, ending the iteration, clipping the prediction box in R 3,t to obtain a single character graph, storing it in the set G, and taking the trained target detection network M1 as a segmentation model;

[0051] S3, constructing a handwritten Chinese text recognition network comprising a segmentation module, a recognition module, and a regular module connected in turn;

[0052] The recognition module adopts FCN, which is used to input the single character graph in the set G, output the predicted character category, and arrange it into an initial text according to the coordinates of the positioning box;

[0053] The regularization module is a BERT model pre-trained on MLM and NSP tasks, used to correct errors in the initial text and generate the final text.

[0054] S4. Use D1 to train the handwritten Chinese text recognition network. In each training session, calculate the localization loss of the segmentation module, the classification loss of the recognition module, and the semantic loss of the regularization module. The three constitute the total loss. Adjust the handwritten Chinese text recognition network to minimize the total loss to obtain the handwritten Chinese text recognition model.

[0055] S5: Obtain the handwritten Chinese text to be recognized, and obtain the corresponding final text through the handwritten Chinese text recognition model.

[0056] The handwritten Chinese datasets include the offline handwritten Chinese dataset CASIA-HWDB, the online handwritten Chinese dataset CASIA-OLHWDB, the ICDAR2013 competition test set, the complex scene handwritten Chinese dataset SCUT-HCCDoc, and the scene Chinese dataset ReCTS.

[0057] The values ​​of θ1 are 0.4~0.5 and θ2 are 0.8~0.9.

[0058] In Sa3, for R 2,t Inner prediction box B t The update method is as follows:

[0059] If B t The predicted bounding box B generated in the (t-1)th iteration t-1 There is overlap, calculate B. t confidence level p t and B t-1 confidence level p t-1 weighted average ,use Update p t ;

[0060] Compare and p t-1 ,like >p t-1 Then B t Otherwise, use B. t-1 Update B t .

[0061] Example 2: See Figures 1 to 7 Regarding the segmentation module, let's take a handwritten text image as an example. The original image is as follows: Figure 3As shown in (a), since the segmentation module first needs to use the pre-trained object detection network M1, during its pre-training, the labeled data in the dataset contains "及", "时", "勉", "岁", "不", ".", so the object detection network M1 can accurately locate these characters, output their prediction boxes and confidence levels. The prediction boxes are as shown in Figure 3 the red boxes in (b), and the confidence levels of these prediction boxes are generally relatively high, higher than the second threshold θ2. However, in addition to being able to generate these prediction boxes relatively accurately, the object detection network M1 also generates many other prediction boxes as shown in Figure 3 (c). Each prediction box has a different confidence level. We label the boxes with a confidence level less than θ1 as yellow, the boxes with a confidence level greater than θ2 as red, and the boxes with a confidence level between θ1 and θ2 as blue. Then, M1 is trained according to S23, which specifically includes:

[0062] The first training:

[0063] Sa1, input the Figure 3 image in (a) into M1 to obtain the localization results of each character, and calculate the localization loss L loc to minimize L loc and adjust the network parameters of M1;

[0064] Sa2, generate the first region R 1,1 , the second region R 2,1 , and the third region R 3,1 for the first training on the sample X. R 1,1 is as shown in Figure 3 the yellow box region in (c), R 2,1 is as shown in Figure 3 the blue region in (c), and the third region R​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​The predicted bounding boxes and confidence levels of each blue box in (d). We randomly select one predicted bounding box corresponding to the character "月" (month), which corresponds to a predicted bounding box and confidence level in Figure 3 (c), and also corresponds to a predicted bounding box and confidence level in Figure 3 (d). Observing that the two predicted bounding boxes overlap, we calculate the weighted average of the two predicted bounding boxes , and use to update p2; then compare and p1. If > p1, we retain the predicted bounding box in Figure 3 (d), otherwise we use the predicted bounding box in Figure 3 (c) to update the predicted bounding box of the character "月" in Figure 3 (d); then compare and θ2. If > θ2, we update the predicted bounding box of the character "月" to R 3,t , that is, change it to a red bounding box as shown in Figure 3 (e), otherwise we do not process it;

[0069] The 3rd to the Tth training is the same as the 2nd. When t = T, the iteration ends. We crop the predicted bounding boxes within R 3,t to obtain single - character images, which are stored in the set G. At this time, character classification and recognition have not been performed. The single - character images are the predicted bounding boxes containing these characters, as shown by the red bounding box in Figure 4~Figure 7 (e).

[0070] We sequentially send the single - character images to the recognition module for recognition. The recognition results are "及", "时", "当", "勉", "厉", "力", "岁", "月", "不", "待", "人", "。". Then, we arrange them according to the coordinates of the positioning bounding boxes to obtain the initial text "及时当勉厉力岁月不待人。". Finally, we send it to the regularization module for error correction. The regularization module has been pre - trained, can understand the context semantics, correct "厉力" to "励", and add punctuation marks to generate the final text "及时当勉励,岁月不待人。".

[0071] Based on the above method, we obtain four handwritten text images, which are respectively labeled as Image 1 - Image 4. The final texts obtained by using the method of the present invention are as shown in Figure 4~Figure 7 . Evaluation index Among them, the first row is the original handwritten text image, the second row is the schematic diagram of the predicted bounding boxes generated by the segmentation module for the original image, and the third row is the final text output by the regularization module.

[0072] Example 3: To illustrate the effect of the present invention, we conduct a comparative experiment using the present invention and two existing technologies, and obtain Table 1:

[0073] Dataset: A mixed dataset composed of CASIA - HWDB and ICDAR2013;

[0074] Comparison method: the present application, CTC recognition algorithm, attention mechanism recognition algorithm;

[0075] Evaluation index: character error rate CER, reasoning speed, annotation cost;

[0076] Table 1. Comparison table of experimental results of different methods on mixed data set

[0077] The present invention CTC recognition algorithm Attention control recognition algorithm Character error rate CER Inference speed (ms) 3.2% 5.8% 4.9% Annotation cost 2.9 16.0 21.5 Text line level Text line level Text line level ​

[0078] From table 1, the character error rate CER of the present application is reduced by 27.6% compared with CTC, the reasoning speed is increased by 5 times, and the annotation cost is reduced by 90%.

[0079] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for recognizing handwritten Chinese text, characterized in that, Comprise the following steps: S1, obtain a handwritten Chinese data set D1, wherein the sample is a handwritten text picture, and the characters in the sample are randomly partially labeled; S2, construct a segmentation module, comprising S21-S24; S21, obtain a target detection network M1, comprising a pre-trained ResNet-18 and FCN, for extracting image features of handwritten text through ResNet-18, and predicting the positioning result of each character in the handwritten text through FCN, the positioning result comprising a prediction box and a corresponding confidence; S22, preset the iteration number T, the first threshold θ1, the second threshold θ2, 0<θ1<θ2<1, and a set G; S23, train M1, wherein the t-th training comprises Sa1-Sa3, 1≤t≤T; Sa1, input sample X into M1 to get the positioning result of each character, calculate the positioning loss L loc to minimize L loc adjust the network parameters of M1; Sa2, generate the first region R of the t-th training on the sample X 1,t , the second region R 2,t , the third region R 3,t , specifically, the prediction box range with the confidence less than θ1 constitutes R 1,t , the prediction box range with the confidence greater than θ2 constitutes R 3,t , the rest of the prediction box range constitutes R 2,t ; Sa3, if t = 1, complete this training, if t > 1, update R in turn 2,t Intra prediction frame and confidence, to R 2,t Intra prediction frame B t , if the updated confidence is greater than θ2, B t Update to R 3,t ; S24, when t = T, iteration ends, clipping R 3,t The inner prediction frame obtains a single word picture, stores it in a set G, and uses the trained target detection network M1 as a segmentation model; S3, construct a handwritten Chinese text recognition network, comprising a segmentation module, a recognition module and a regular module connected in turn; The recognition module uses FCN to input single word images in set G, output predicted character categories, and arrange them into initial text according to the coordinates of the positioning box; The regular module is a BERT model pre-trained through MLM task and NSP task, used to correct the initial text to generate the final text; S4, train the handwritten Chinese text recognition network with D1, calculate the positioning loss of the segmentation module, the classification loss of the recognition module and the semantic loss of the regular module in each training, and adjust the handwritten Chinese text recognition network to minimize the total loss to obtain a handwritten Chinese text recognition model; S5, obtain a handwritten Chinese text to be recognized, and obtain the corresponding final text through the handwritten Chinese text recognition model.

2. The method for recognizing handwritten Chinese text according to claim 1, wherein, The handwritten Chinese data set comprises an offline handwritten Chinese data set CASIA-HWDB, an online handwritten Chinese data set CASIA-OLHWDB, an ICDAR2013 competition test set, a complex scene handwritten Chinese data set SCUT-HCCDoc and a scene Chinese data set ReCTS.

3. The method of claim 1, wherein, θ1 is 0.4-0.5, and θ2 is 0.8-0.

9.

4. The method for recognizing handwritten Chinese text according to claim 1, wherein, In Sa3, R 2,t a prediction block B t The updating method is: If B t the prediction frame B t-1 has overlap, calculate B t the confidence p t of B t-1 the confidence p t-1 of B , the weighted average of p , update p t ; compare and p t-1 if > p t-1 then B t unchanged, otherwise update B t-1 with B t .

Images