Online handwritten chinese text line recognition method based on local feature map real-time update

Abstract

This invention proposes an online handwritten Chinese text line recognition method based on real-time updating of local feature maps, comprising the following steps: Step S1, initializing the real-time recognition model; Step S2, when starting stroke input, taking the touch of the pen tip on the handwriting surface of the handwriting device as the start marker; Step S3, acquiring the trajectory of the online handwritten Chinese text line, recording the coordinate point sequence of the strokes, and taking the lifting of the pen as the end marker of a stroke; Step S4, extracting the local image corresponding to the newly input stroke and performing preprocessing; Step S5, calculating the CNN features of the local image corresponding to the newly input stroke, replacing the new features on the previous feature map, and realizing real-time updating of the local features of the new stroke; Step S6, using a language model to decode and update the recognition result, realizing real-time text line recognition. This invention significantly improves speed, has low image size dependence, strong scalability and robustness, and can be used in various resource-constrained embedded devices, possessing high application value.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and document image analysis technology, and in particular to an online handwritten Chinese text line recognition method based on real-time updates of local feature maps. Background Technology

[0002] Handwritten text recognition, as a crucial part of Optical Character Recognition (OCR), has always been a key research focus for many researchers. Online handwritten Chinese text line recognition is more complex than printed text recognition, facing challenges such as significant variations in font styles and a wide variety of font types. Currently, most recognition methods are based on image encoding and decoding, which have achieved some success. However, for Chinese text, especially long lines, the generated images are too large to smoothly achieve real-time recognition of online handwritten text. For users, waiting for the entire sentence to be written before clicking to recognize it requires a click and a short wait, which is unacceptable. In practical use, to avoid lag, applications are often limited to recognizing single characters or short texts, which restricts the system's practicality. Most current real-time recognition methods use trajectory-based over-segmentation methods for text line recognition. These methods primarily rely on LSTM for recognition, but due to the large number of long lines in online handwritten Chinese text, the trajectory sequences are very long, making accurate over-segmentation difficult using LSTM methods, and thus hindering the achievement of good recognition accuracy. With the development of deep learning, image-based non-segmentation recognition methods have gradually become dominant in text line recognition models in recent years, offering certain advantages for recognizing long text lines. However, in order to preserve complete information, large text line images need to be generated in online handwritten text line recognition. If image-based text line recognition methods are used for real-time recognition, it will take a lot of time, affecting the user's writing experience and hindering the application of real-time recognition in practical scenarios such as embedded devices. Summary of the Invention

[0003] This invention proposes an online handwritten Chinese text line recognition method based on real-time updates of local feature maps. It significantly improves speed, has low image size dependence, strong scalability and robustness, and can be used in various resource-constrained embedded devices, thus having high application value.

[0004] The present invention adopts the following technical solution.

[0005] A method for online handwritten Chinese text line recognition based on real-time updates of local feature maps is used for real-time recognition of handwritten text, including the following steps;

[0006] Step S1: Initialize the real-time recognition model;

[0007] Step S2: When starting stroke input, the starting point is when the pen tip touches the handwriting surface of the handwriting device;

[0008] Step S3: Collect the line trajectory of online handwritten Chinese text, record the coordinate point sequence of each stroke, and use the lifting of the pen as a marker for the end of a stroke;

[0009] Step S4: Extract the local image corresponding to the newly input stroke and preprocess it;

[0010] Step S5: Calculate the CNN features of the local image corresponding to the new input stroke, and replace the new features with the previous feature map to realize the real-time update of the local features of the new stroke.

[0011] Step S6: Use CTC combined with an n-gram language model to decode and update the recognition results to achieve real-time recognition of online handwritten Chinese text lines.

[0012] In step S1, the real-time recognition model includes a deep convolutional neural network encoder, an RNN decoder, and a CTC and language model transcription layer. During initialization, the real-time recognition model first initializes a text line image with a preset height and width and all pixel values ​​of zero. Then, the text line image is fed into the handwritten text line recognition model to obtain the global output feature map of each convolutional layer and pooling layer in the handwritten text line recognition model, which is used for subsequent real-time feature updates.

[0013] The height and width of the text line image match the actual handwriting scene. The handwritten text is written from left to right and allows reverse pen insertion. The size of the handwritten text matches the size of the text line image.

[0014] In step S1, the real-time recognition model extracts features from the image based on CRNN. ​​The pooling layers in the network are all set to average pooling. Except for the first two convolutional layers, all other convolutional layers are followed by batch normalization layers.

[0015] The real-time recognition model uses 7 convolutional layers and 4 pooling layers to extract features. The number of convolutional kernels corresponds to the number of output feature maps. The number of convolutional kernels in the 7 convolutional layers are {64, 128, 256, 256, 256, 512, 512}. Finally, a global average pooling layer is used to pool the image to a height of 1. The width remains unchanged during the global average pooling process.

[0016] The real-time recognition model uses a two-layer stacked bidirectional LSTM RNN with 256 hidden nodes. Finally, CTC and Beam Search are used to introduce the language model for final decoding.

[0017] In step S3, the trajectory of the online handwritten Chinese text is collected, and the coordinate point sequence of each stroke is recorded and represented as (x1,y1),(x2,y2),...,(xn,yn), with the lifting of the pen as the end mark of a stroke.

[0018] In step S4, the local image corresponding to the newly input stroke is extracted and preprocessed; specifically: when the user inputs a new stroke, the left and right x-axis boundaries of the stroke are first calculated, denoted as L and R. The convolution kernel size is set to 3×3, and two columns of pixels are added to each side of the boundary. Thus, the size of the extracted local image M is:

[0019]

[0020] Where H and W are the height and width of the image, respectively; the top two rows on the right of Formula 1 correspond to the first and second cases, respectively, where the stroke boundary is on the left and right edges of the original image; the third case corresponding to the third row on the right of Formula 1 is a rectangular area composed of two columns of pixels on each of the left and right edges of the local image after the stroke is written.

[0021] The first case is at the left edge of the original image (L≤1). In this case, the right edge is expanded by 2 columns of pixels, and the left edge is expanded to the first column on the left. Therefore, the width is ((W-1)-(L-2)).

[0022] The second case is at the right edge of the original image (R≥W-2). In this case, the left boundary is expanded by 2 columns of pixels, and the right boundary is expanded to the last column on the right. Therefore, the width is ((R+2)-0).

[0023] After cropping a local image, padding is applied to the cropped local image M. Specifically, padding is applied to the top and bottom edges of the local image M by one row of pixels to simulate the padding required for convolution in the original image. If the local image M is on the left or right sides of the original image, padding is applied according to different situations, expressed by the formula:

[0024]

[0025] In the formula, P L P R These refer to the padding on the left and right sides, respectively. If the local image is not on the left or right sides of the original image, then the local image has already been expanded by 2 columns of pixels on the left and right sides, and no padding operation is needed, i.e., P L =P R =0; If the local image is on the left or right side of the original image, such as the left side (L=0), then the left padding column needs to be all 0 pixels to simulate the padding operation of the original image.

[0026] In step S5, only the CNN features of the local image corresponding to the new input stroke are calculated, and the new features are used to replace the previous feature map to achieve real-time updating of the local features of the new stroke. The specific method is as follows:

[0027] The captured local image is subjected to local convolution, iteratively convolving, pooling, and updating the feature map of each network layer. No padding is applied during pooling. The number of pixels to be expanded during convolution and the padding are as follows:

[0028]

[0029]

[0030] W i The width of the i-th layer feature map is represented by E(L, R). E(L, R) represents the number of pixels to be expanded. When it is (-2, +2), that is, 2 columns are expanded on the left and right sides, the corresponding pixel values ​​of the original feature map are moved to the local feature map to be updated, and the iterative convolution and feature map update of the next layer continue.

[0031] When expanding the feature map pixels, consider whether it is divisible by the pooling layer stride. Specifically: when the pooling layer stride is 2, if the current L is not divisible by stride 2, then expand to the left by L%2 columns of pixels; if it is expanded to the stride S, then expand to the left by L%S columns of pixels, considering the current Lth column; similarly, for the right side of the local feature, then expand to the right by SL%S columns of pixels, considering the current Rth column.

[0032] In step S6, CTC is used in conjunction with the N-Gram language model to decode and update the recognition results, thereby achieving real-time recognition of online handwritten Chinese text lines.

[0033] This invention proposes a real-time online handwritten Chinese text line recognition method based on real-time updates of local feature maps. For online handwritten Chinese text line recognition, an improved CRNN is used for image feature extraction and decoding, achieving real-time recognition of online handwritten Chinese text lines during the inference phase. This method achieves excellent results in two different types of handwriting scenarios, with a significant speed improvement, low image size dependence, strong scalability and robustness. This method can be used in various resource-constrained embedded devices and has high application value.

[0034] The present invention significantly improves the recognition speed, has low image size dependence, strong scalability and robustness, and can be used in various resource-constrained embedded devices, thus having high application value.

[0035] Compared with the prior art, the present invention also has the following beneficial effects:

[0036] (1) Implement online handwritten Chinese text line recognition.

[0037] (2) Real-time recognition of online handwritten Chinese text lines.

[0038] (3) The method proposed in this invention has high scalability and can be applied to other network structures or even other additional features such as path re-integral features. Attached Figure Description

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0040] Appendix Figure 1 This is a flowchart of the method described in this invention;

[0041] Appendix Figure 2 This is a schematic diagram of the overall network structure for online handwritten text line recognition;

[0042] Appendix Figure 3 This is a schematic diagram of the partially updated area;

[0043] Appendix Figure 4 This is a schematic diagram illustrating an implementation example of the method. Detailed Implementation

[0044] As shown in the figure, an online handwritten Chinese text line recognition method based on real-time updating of local feature maps is used for real-time recognition of handwritten text, including the following steps;

[0045] Step S1: Initialize the real-time recognition model;

[0046] Step S2: When starting stroke input, the starting point is when the pen tip touches the handwriting surface of the handwriting device;

[0047] Step S3: Collect the line trajectory of online handwritten Chinese text, record the coordinate point sequence of each stroke, and use the lifting of the pen as a marker for the end of a stroke;

[0048] Step S4: Extract the local image corresponding to the newly input stroke and preprocess it;

[0049] Step S5: Calculate the CNN features of the local image corresponding to the new input stroke, and replace the new features with the previous feature map to realize the real-time update of the local features of the new stroke.

[0050] Step S6: Use CTC combined with an n-gram language model to decode and update the recognition results to achieve real-time recognition of online handwritten Chinese text lines.

[0051] In step S1, the real-time recognition model includes a deep convolutional neural network encoder, an RNN decoder, and a CTC and language model transcription layer. During initialization, the real-time recognition model first initializes a text line image with a preset height and width and all pixel values ​​of zero. Then, the text line image is fed into the handwritten text line recognition model to obtain the global output feature map of each convolutional layer and pooling layer in the handwritten text line recognition model, which is used for subsequent real-time feature updates.

[0052] The height and width of the text line image match the actual handwriting scene. The handwritten text is written from left to right and allows reverse pen insertion. The size of the handwritten text matches the size of the text line image.

[0053] In step S1, the real-time recognition model extracts features from the image based on CRNN. ​​The pooling layers in the network are all set to average pooling. Except for the first two convolutional layers, all other convolutional layers are followed by batch normalization layers.

[0054] The real-time recognition model uses 7 convolutional layers and 4 pooling layers to extract features. The number of convolutional kernels corresponds to the number of output feature maps. The number of convolutional kernels in the 7 convolutional layers are {64, 128, 256, 256, 256, 512, 512}. Finally, a global average pooling layer is used to pool the image to a height of 1. The width remains unchanged during the global average pooling process.

[0055] The real-time recognition model uses a two-layer stacked bidirectional LSTM RNN with 256 hidden nodes. Finally, CTC and Beam Search are used to introduce the language model for final decoding.

[0056] In step S3, the trajectory of the online handwritten Chinese text is collected, and the coordinate point sequence of each stroke is recorded and represented as (x1, y1), (x2, y2), ..., (xn, yn), with the lifting of the pen as the end mark of a stroke.

[0057] In step S4, the local image corresponding to the newly input stroke is extracted and preprocessed; specifically: when the user inputs a new stroke, the left and right x-axis boundaries of the stroke are first calculated, denoted as L and R. The convolution kernel size is set to 3×3, and two columns of pixels are added to each side of the boundary. Thus, the size of the extracted local image M is:

[0058]

[0059] Where H and W are the height and width of the image, respectively; the top two rows on the right of Formula 1 correspond to the first and second cases, respectively, where the stroke boundary is on the left and right edges of the original image; the third case corresponding to the third row on the right of Formula 1 is a rectangular area composed of two columns of pixels on each of the left and right edges of the local image after the stroke is written.

[0060] The first case is at the left edge of the original image (L≤1). In this case, the right edge is expanded by 2 columns of pixels, and the left edge is expanded to the first column on the left. Therefore, the width is ((W-1)-(L-2)).

[0061] The second case is at the right edge of the original image (R≥W-2). In this case, the left boundary is expanded by 2 columns of pixels, and the right boundary is expanded to the last column on the right. Therefore, the width is ((R+2)-0).

[0062] After cropping a local image, padding is applied to the cropped local image M. Specifically, padding is applied to the top and bottom edges of the local image M by one row of pixels to simulate the padding required for convolution in the original image. If the local image M is on the left or right sides of the original image, padding is applied according to different situations, expressed by the formula:

[0063]

[0064] In the formula, P L P R These refer to the padding on the left and right sides, respectively. If the local image is not on the left or right sides of the original image, then the local image has already been expanded by 2 columns of pixels on the left and right sides, and no padding operation is needed, i.e., P L =P R =0; If the local image is on the left or right side of the original image, such as the left side (L=0), then the left padding column needs to be all 0 pixels to simulate the padding operation of the original image.

[0065] In step S5, only the CNN features of the local image corresponding to the new input stroke are calculated, and the new features are used to replace the previous feature map, thus achieving real-time updating of the local features of the new stroke. The specific method is as follows:

[0066] The captured local image is subjected to local convolution, iteratively convolving, pooling, and updating the feature map of each network layer. No padding is applied during pooling. The number of pixels to be expanded during convolution and the padding are as follows:

[0067]

[0068]

[0069] W iThe width of the i-th layer feature map is represented by E(L, R). E(L, R) represents the number of pixels to be expanded. When it is (-2, +2), that is, 2 columns are expanded on the left and right sides, the corresponding pixel values ​​of the original feature map are moved to the local feature map to be updated, and the iterative convolution and feature map update of the next layer continue.

[0070] When expanding the feature map pixels, consider whether it is divisible by the pooling layer stride. Specifically: when the pooling layer stride is 2, if the current L is not divisible by stride 2, then expand to the left by L%2 columns of pixels; if it is expanded to the stride S, then expand to the left by L%S columns of pixels, considering the current Lth column; similarly, for the right side of the local feature, then expand to the right by SL%S columns of pixels, considering the current Rth column.

[0071] In step S6, CTC is used in conjunction with the N-Gram language model to decode and update the recognition results, thereby achieving real-time recognition of online handwritten Chinese text lines.

[0072] In this example, step S1 initializes a line of all-black text with a height of 128 and a width of 2400.

[0073] In this example, CTC is a text recognition algorithm, and N-Gram is a language model commonly used in large-vocabulary continuous speech recognition. For Chinese, it can be called the Chinese Language Model (CLM).

[0074] ——————————————————————————

Claims

1. An online handwritten Chinese text line recognition method based on real-time updating of local feature maps, used for real-time recognition of handwritten text, characterized in that: Includes the following steps; Step S1: Initialize the real-time recognition model; Step S2: When starting stroke input, the starting point is when the pen tip touches the handwriting surface of the handwriting device; Step S3: Collect the line trajectory of online handwritten Chinese text, record the coordinate point sequence of each stroke, and use the lifting of the pen as a marker for the end of a stroke; Step S4: Extract the local image corresponding to the newly input stroke and preprocess it; Step S5: Calculate the CNN features of the local image corresponding to the new input stroke, and replace the new features with the previous feature map to realize the real-time update of the local features of the new stroke. Step S6: Use CTC combined with an n-gram language model to decode and update the recognition results to achieve real-time recognition of online handwritten Chinese text lines; In step S1, the real-time recognition model includes a deep convolutional neural network encoder, an RNN decoder, and a CTC and language model transcription layer. During initialization, the real-time recognition model first initializes a text line image with a preset height and width and all pixel values ​​of zero. Then, the text line image is fed into the handwritten text line recognition model to obtain the global output feature map of each convolutional layer and pooling layer in the handwritten text line recognition model, which is used for subsequent real-time feature updates. The height and width of the text line image match the actual handwriting scene. The handwritten text is written from left to right and allows reverse pen insertion. The size of the handwritten text matches the size of the text line image. In step S1, the real-time recognition model extracts image features based on CRNN. ​​The pooling layers in the network are all set to average pooling. Except for the first two convolutional layers, all other convolutional layers are followed by batch normalization layers. The real-time recognition model uses a two-layer stacked bidirectional LSTM in the RNN part, and finally introduces a language model through CTC and Beam Search for final decoding. That is, CRNN is used to extract and decode image features, and in the inference stage, real-time recognition of online handwritten Chinese text lines is achieved.

2. The online handwritten Chinese text line recognition method based on real-time updating of local feature maps according to claim 1, characterized in that: The real-time recognition model uses 7 convolutional layers and 4 pooling layers to extract features. The number of convolutional kernels corresponds to the number of output feature maps. The number of convolutional kernels in the 7 convolutional layers are {64, 128, 256, 256, 256, 512, 512}. Finally, a global average pooling layer is used to pool the image to a height of 1. The width remains unchanged during the global average pooling process. The real-time recognition model uses a two-layer stacked bidirectional LSTM for its RNN part, with 256 hidden layer nodes.

3. The online handwritten Chinese text line recognition method based on real-time updating of local feature maps according to claim 1, characterized in that: In step S3, the trajectory of the online handwritten Chinese text is collected, and the coordinate point sequence of each stroke is recorded and represented as (x1,y1), (x2,y2), ..., (xn,yn), with the lifting of the pen as the end mark of a stroke.

4. The online handwritten Chinese text line recognition method based on real-time updating of local feature maps according to claim 1, characterized in that: In step S4, the local image corresponding to the newly input stroke is extracted and preprocessed; specifically: when the user inputs a new stroke, the left and right x-axis boundaries of the stroke are first calculated, denoted as L and R. The convolution kernel size is set to 3×3, and two columns of pixels are added to each side of the boundary. The resulting extracted local image M is of the following size: Where H and W are the height and width of the image, respectively; the top two rows on the right of Formula 1 correspond to the first and second cases, respectively, where the stroke boundary is on the left and right edges of the original image; the third case corresponding to the third row on the right of Formula 1 is a rectangular area composed of two columns of pixels on each of the left and right edges of the local image after the stroke is written. The first case is at the left edge of the original image, where L≤1. In this case, the right boundary is expanded by 2 columns of pixels, and the left boundary is expanded to the first column on the left. Therefore, the width is ((W-1)-(L-2)). The second case is on the right edge of the original image, where R ≥ W - 2. In this case, the left boundary is expanded by 2 columns of pixels, and the right boundary is expanded to the last column on the right. Therefore, the width is ((R+2)-0).

5. The online handwritten Chinese text line recognition method based on real-time updating of local feature maps according to claim 4, characterized in that: After cropping a local image, padding is applied to the cropped local image M. Specifically, padding is applied to the top and bottom edges of the local image M by one row of pixels to simulate the padding required for convolution in the original image. If the local image M is on the left or right sides of the original image, padding is applied according to different situations, expressed by the formula: In the formula, P L P R These are the padding cases for the left and right sides, respectively. If the local image is not on the left or right sides of the original image, then the local image has already been expanded by 2 columns of pixels on both sides, and no padding operation is needed, i.e., P L =P R =0; If the local image is on the left or right side of the original image, and the left side L=0, then the left padding column needs to be all 0 pixels to simulate the padding operation of the original image.

6. The online handwritten Chinese text line recognition method based on real-time updating of local feature maps according to claim 5, characterized in that: In step S5, only the CNN features of the local image corresponding to the new input stroke are calculated, and the new features are used to replace the previous feature map to achieve real-time updating of the local features of the new stroke. The specific method is as follows: The captured local image is subjected to local convolution, iteratively convolving, pooling, and updating the feature map of each network layer; no padding is applied during pooling, and the pixels to be expanded during convolution and the padding are as follows: W i The width of the i-th layer feature map is represented by E(L,R); E(L,R) represents the number of pixels to be expanded. When it is (-2,+2), that is, 2 columns are expanded on the left and right sides, the corresponding pixel values ​​of the original feature map are moved to the local feature map to be updated, and the iterative convolution and feature map update of the next layer continue. When expanding the feature map pixels, consider whether it is divisible by the pooling layer stride. Specifically: when the pooling layer stride is 2, if the current L is not divisible by stride 2, then expand to the left by L%2 columns of pixels; if it is expanded to the stride S, then expand to the left by L%S columns of pixels, considering the current Lth column; similarly, for the right side of the local feature, then expand to the right by SL%S columns of pixels, considering the current Rth column.

7. The online handwritten Chinese text line recognition method based on real-time updating of local feature maps according to claim 1, characterized in that: In step S6, CTC is used in conjunction with the N-Gram language model to decode and update the recognition results, thereby achieving real-time recognition of online handwritten Chinese text lines.

Images