A scene text recognition method based on a convolutional neural network model

Abstract

The application discloses a scene text recognition method based on a convolutional neural network model, and comprises the following steps: using a feature encoder to preliminarily extract features of a text picture; performing bilinear upsampling or maximum pooling processing on the extracted feature map, so that features of different dimensions are converted into the same dimension; constructing an RNSTR network model, training the RNSTR network model by using a training set; and inputting a to-be-tested picture, and recognizing the to-be-tested picture by using the constructed RNSTR network model. The application solves the problem of low scene text recognition accuracy, and realizes automatic recognition of scene text.

Description

Technical Field

[0001] This invention relates to the field of computer image processing, and in particular to a method for scene text recognition based on a convolutional neural network model. Background Technology

[0002] Scene text recognition, as an important research direction in computer vision, has received increasing attention. Scene text recognition involves detecting and recognizing text images in natural scenes, and it has broad research value and application prospects in fields such as industrial automation, autonomous driving, and image retrieval. However, in real-world natural scene contexts, text images often suffer from problems such as text deformation, complex font styles, and varying character scales. These issues significantly increase the difficulty of automatic scene text recognition, making effective scene text recognition challenging.

[0003] Most current scene text recognition methods involve two main steps: detecting text in an image and recognizing the detected text. Traditional scene text recognition methods are generally inefficient, have low accuracy, and weak anti-interference capabilities, failing to meet the needs of practical engineering. With the continuous development of deep learning technology, many excellent neural network-based algorithms have emerged in recent years. Common methods include Support Vector Machines (SVM), Modified Quadratic Discriminant Functions (MQDF), and Discriminative Learning Quadratic Discriminant Functions (DLQDF). However, the recognition accuracy of these methods is significantly lower than human performance, failing to meet practical requirements.

[0004] Therefore, LeCun proposed Convolutional Neural Networks (CNNs), which achieved good recognition results and are currently the main method for scene text recognition. For example, Shi et al. first used a combination of CNNs and recurrent neural networks to achieve end-to-end scene text recognition. Lee and Bahdanau et al., with a framework structure shared with CNNs and recurrent neural networks, incorporated the attention mechanism, which has emerged in the field of machine translation, into the field of scene text recognition, achieving good scene text recognition results. Summary of the Invention

[0005] This invention provides a scene text recognition method based on a convolutional neural network model. This invention solves the problem of low accuracy in scene text recognition and achieves automatic scene text recognition, as detailed below:

[0006] A scene text recognition method based on a convolutional neural network model, the method comprising:

[0007] A feature encoder is used to perform preliminary feature extraction on the text image;

[0008] The extracted feature maps are subjected to bilinear upsampling or max pooling to transform features of different dimensions into the same dimension.

[0009] Construct an RNSTR network model and train it using the training set; input the image to be tested and use the constructed RNSTR network model for recognition.

[0010] The RNSTR network model consists of three parts: a spatial location information perception module, a global feature enhancement module, and a feature fusion module.

[0011] The spatial location information perception module includes 10 convolutional layers, 5 activation layers, and 1 pooling layer. The feature fusion includes a convolutional layer C3_1, an adaptive average pooling layer AP3_1, and an activation layer RL3_1.

[0012] Furthermore, the global feature enhancement consists of two convolutional layers and one activation layer.

[0013] The beneficial effects of the technical solution provided by this invention are:

[0014] 1. This invention improves upon the classic convolutional neural network model and proposes an RNSTR model, which enables automatic recognition of scene text;

[0015] 2. This invention designs a dual-path parallel convolutional neural network structure; the two branches extract spatial feature information between different dimension feature layers of text data respectively, and selectively enhance the learned feature information to obtain a feature image with high correlation;

[0016] 3. This invention improves the convolutional neural network by preserving the spatial details and texture features contained in low-dimensional features, while extracting the contextual semantic information contained in high-dimensional spatial features. Attached Figure Description

[0017] Figure 1 This is a flowchart of a scene text recognition method based on a convolutional neural network model;

[0018] Figure 2 This is a schematic diagram of the RNSTR model network structure;

[0019] Figure 3 This is a schematic diagram of the original convolutional feature extraction module.

[0020] Figure 4 This is a schematic diagram of the improved spatial position sensing module.

[0021] Table 1 compares the recognition accuracy of several methods on the test dataset. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0023] Example 1

[0024] This invention provides a scene text recognition method based on a convolutional neural network model. See [link to relevant documentation]. Figure 1 The method includes the following steps:

[0025] 101: Use a feature encoder to perform preliminary feature extraction on the text image;

[0026] 102: Perform bilinear upsampling or max pooling on the extracted feature maps to transform features of different dimensions into the same dimension;

[0027] 103: Construct an RNSTR network model and train it using the training set; input the image to be tested and use the constructed RNSTR network model for recognition.

[0028] The deep learning framework used in this embodiment of the invention is PyTorch. [1] An RNSTR network model was constructed based on the PyTorch framework. This network model consists of three parts: a spatial location information perception module, a global feature enhancement module, and a feature fusion module. The spatial location information perception module includes 10 convolutional layers and 5 activation layers.

[0029] C1_1 is a convolutional layer with a kernel size of 3×3, a stride of 1, an input image size of 8×25, an output image size of 8×25, and 32 channels.

[0030] C1_2 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 8×25, an output image size of 8×25, and a channel count of 1.

[0031] SM1_1 is the activation layer, using softmax as the activation function. The input image size is 8×25, and the output image size is 8×25.

[0032] C1_3 is a convolutional layer with a kernel size of 3×3, a stride of 1, an input image size of 8×25, an output image size of 8×25, and 32 channels.

[0033] C1_4 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 8×25, an output image size of 8×25, and a channel size of 1.

[0034] SM1_2 is the activation layer, using softmax as the activation function. The input image size is 8×25, and the output image size is 8×25.

[0035] C1_5 is a convolutional layer with a kernel size of 3×3, a stride of 1, an input image size of 8×25, an output image size of 8×25, and 64 channels.

[0036] C1_6 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 8×25, an output image size of 8×25, and a channel count of 1.

[0037] SM1_3 is the activation layer, using softmax as the activation function. The input image size is 8×25, and the output image size is 8×25.

[0038] C1_7 is a convolutional layer with a kernel size of 3×3, a stride of 1, an input image size of 8×25, an output image size of 8×25, and 128 channels.

[0039] C1_8 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 8×25, an output image size of 8×25, and a channel count of 1.

[0040] SM1_4 is the activation layer, using softmax as the activation function. The input image size is 8×25, and the output image size is 8×25.

[0041] C1_9 is a convolutional layer with a kernel size of 3×3, a stride of 1, an input image size of 8×25, an output image size of 8×25, and 256 channels.

[0042] C1_10 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 8×25, an output image size of 8×25, and a channel count of 1.

[0043] SM1_5 is the activation layer, using softmax as the activation function. The input image size is 8×25, and the output image size is 8×25.

[0044] MP_1 is a pooling layer with a pooling filter size of 2×2, a stride of 0, an input image size of 16×50, and an output image size of 8×25.

[0045] The structure of the feature fusion part is as follows:

[0046] C3_1 is a convolutional layer with a kernel size of 3×3, a stride of 1, an input image size of 8×25, an output image size of 8×25, and 512 channels.

[0047] AP3_1 is an adaptive average pooling layer with an input image size of 8×25 and an output image size of 1×1.

[0048] RL3_1 is the activation layer, using Sigmoid as the activation function. The input image size is 1×1, and the output image size is 1×1.

[0049] In summary, the embodiments of the present invention improve the convolutional neural network by preserving the spatial details and texture features contained in low-dimensional features, while extracting the contextual semantic information contained in high-dimensional spatial features.

[0050] Example 2

[0051] The global feature enhancement part of Example 1 and the RNSTR network model will be further introduced below with specific examples. See the description below for details:

[0052] 201: To combine high-level semantic information with low-level image information, this embodiment of the invention designs a global feature enhancement structure for the RNSTR model, which consists of two convolutional layers and one activation layer. The output obtained from the encoder is first concatenated along the channel dimension, allowing feature extraction from different scales of a scene text image to more comprehensively reflect image information. The global feature enhancement structure proposed in this embodiment can learn the overall feature representation and collect the finest-grained feature representations to suppress useless information, thereby improving the representational power of the features.

[0053] 202: The specific structure of the global feature enhancement part, consisting of two convolutional layers and one activation layer, is as follows:

[0054] C2_1 is a convolutional layer with a kernel size of 3×3, a stride of 1, an input image size of 8×25, an output image size of 8×25, and 496 channels.

[0055] C2_2 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 8×25, an output image size of 8×25, and a channel count of 1.

[0056] SM3_1 is the activation layer, using softmax as the activation function. The input image size is 1×1, and the output image size is 1×1.

[0057] 204: Use the constructed RNSTR model to train the dataset;

[0058] Among them, seven internationally published real STR datasets (IIIT5K-Words (IIIT5K), Street ViewText (SVT), ICDAR2003 (IC03), ICDAR2013 (IC13), ICDAR2015 (IC15), SVT Perspective (SVTP), and CUTE80 (CUTE)) were evaluated, and three synthetic datasets (Synth90K, SynthText, and SynthAdd) were selected for training.

[0059] 205: The image information in the training dataset is used as the input layer and fed into the constructed RNSTR model. The extracted feature maps are then subjected to bilinear upsampling or max pooling to make the images have the same size and to transform features of different dimensions into the same dimension.

[0060] Bilinear upsampling and max pooling are well known to those skilled in the art, and will not be described in detail in this embodiment of the invention.

[0061] 206: After obtaining the trained model, the segmented and processed text character images are input into the trained model for recognition, and the recognized scene text results are obtained.

[0062] In summary, this invention utilizes the proposed spatial location information perception module and global feature enhancement module to design a dual-path parallel convolutional neural network encoder structure. The two branches extract spatial feature information between different dimensional feature layers of text data and selectively enhance the learned feature information to obtain a highly correlated feature image. By fusing the outputs of the two branches, the expressive power of the features can be improved, thereby enhancing the robustness of the feature image and increasing the recognition accuracy. This model is used to achieve automatic recognition of scene text.

[0063] Example 3

[0064] The feasibility of the schemes in Examples 1 and 2 is verified below with reference to Table 1, as detailed in the following description:

[0065] 301: In this embodiment of the invention, seven internationally published real STR datasets (IIIT5K-Words (IIIT5K), Street View Text (SVT), ICDAR2003 (IC03), ICDAR2013 (IC13), ICDAR2015 (IC15), SVT Perspective (SVTP), and CUTE80 (CUTE)) were used for evaluation, and three synthetic datasets (Synth90K, SynthText, and SynthAdd) were selected for training.

[0066] 302: First, the dataset is trained and tested using the constructed RNSTR network model to obtain the trained model;

[0067] In the experiments of this invention embodiment, the operating system was Ubuntu 16.04, the GPU model was GTX1080Ti, the memory was 11G, and the PyTorch deep learning framework was used.

[0068] 303: Analysis of Experimental Results:

[0069] Different recognition methods were used to identify the IIIT5K-Words (IIIT5K), Street View Text (SVT), ICDAR2003 (IC03), ICDAR2013 (IC13), ICDAR2015 (IC15), SVT Perspective (SVTP), and CUTE80 (CUTE) datasets. The recognition accuracy is shown in Table 1.

[0070] As can be seen from Table 1, the RNSTR model designed based on the embodiments of the present invention has a recognition accuracy that is not only significantly better than the convolutional model, but also better than other deep learning-based text recognition methods. This proves that the RNSTR model of the present invention can effectively perform deep processing on the extracted feature information, which helps to improve network performance, and also proves the effectiveness of the model.

[0071] Table 1 Comparison of recognition accuracy (%) of several methods on the test dataset

[0072]

[0073] References

[0074] [1]Paszke, A., Gross, S., Massa, F., Lerer, A., Chintala, S., 2019. Pytorch: Animperative style, high-performance deep learning library.

[0075] [2]He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition, in: IEEE Conference on Computer Vision and Pattern Recognition.

[0076] [3]Shi B,Wang X,Lyu P,Yao C,Bai X(2016)Robust scene text recognitionwith automatic rectifification.In:IEEE Conference on Computer Vision andPattern Recognition,pp 4167-4176.

[0077] [4]Shi,B.,Yang,M.,Wang,X.,Lyu,P.,Yao,C.,Bai,X.,2019.Aster:Anattentional scene text recognizer with flexible rectification.IEEETransactions on Pattern Analysis and Machine Intelligence 41,2035–2048.

[0078] [5]Luo,C.,Jin,L.,Sun,Z.,2019.Moran:A multi-object rectified attentionnetwork for scene text recognition.Pattern Recognition 90.

[0079] [6]Zie,Z.,Huang,Y.,Zhu,Y.,Jin,L.,Liu,Y.,Xie,L.,2019.Aggregationcross-entropy for sequence recognition,in:2019 IEEE / CVF Conference onComputer Vision and Pattern Recognition(CVPR),pp.6531–6540.

[0080] [7]Wan,Z.,He,M.,Chen,H.,Bai,X.,Yao,C.,2020.Textscanner:Readingcharacters in order for robust scene text recognition,in:National Conferenceon Artificial Intelligence.

[0081] [8]Lin,Q.,Luo,C.,Jin,L.,Lai,S.,2021b.Stan:A sequential transformationattention-based network for scene text recognition.Pattern Recognition 111,107692

[0082] [9]Xiao,Z.,Nie,Z.,Song,C.,Chronopoulos,AT,2022.An extendedattention mechanism for scene text recognition.Expert Systems withApplications 203,117377.

[0083]

[10] Li,B,2022.EMU:Effective Multi-Hot Encoding Net for LightweightScene Text Recognition With a Large Character Set.IEEE Transactions onCircuits and Systems for Video Technology 32,5374–5385.

[0084] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A scene text recognition method based on a convolutional neural network model, characterized in that, The method includes: Construct an RNSTR network model and train it using the training set; input the image to be tested and use the trained RNSTR network model to perform scene text recognition. The RNSTR network model adopts a dual-path parallel structure, including a first feature extraction branch, a second feature extraction branch, and a feature fusion module. The first feature extraction branch includes a spatial location information perception module, which is used to extract spatial feature information between different dimensional feature layers; The second feature extraction branch includes a global feature enhancement module, which is used to selectively enhance the features of the image under test; The first feature extraction branch and the second feature extraction branch work in parallel, and their outputs are fused by the feature fusion module. The spatial location information sensing module includes: 10 convolutional layers, 5 activation layers, and 1 pooling layer; The feature fusion module includes: a convolutional layer C3_1, an adaptive average pooling layer AP3_1, and an activation layer RL3_1; The global feature enhancement module consists of two convolutional layers and one activation layer; The 10 convolutional layers, 5 activation layers, and 1 pooling layer are: C1_1 is a convolutional layer with a kernel size of 3×3, a stride of 1, an input image size of 8×25, an output image size of 8×25, and 32 channels. C1_2 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 8×25, an output image size of 8×25, and a channel count of 1. SM1_1 is the activation layer, using softmax as the activation function. The input image size is 8×25, and the output image size is 8×25. C1_3 is a convolutional layer with a kernel size of 3×3, a stride of 1, an input image size of 8×25, an output image size of 8×25, and 32 channels. C1_4 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 8×25, an output image size of 8×25, and a channel size of 1. SM1_2 is the activation layer, using softmax as the activation function. The input image size is 8×25, and the output image size is 8×25. C1_5 is a convolutional layer with a kernel size of 3×3, a stride of 1, an input image size of 8×25, an output image size of 8×25, and 64 channels. C1_6 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 8×25, an output image size of 8×25, and a channel count of 1. SM1_3 is the activation layer, using softmax as the activation function. The input image size is 8×25, and the output image size is 8×25. C1_7 is a convolutional layer with a kernel size of 3×3, a stride of 1, an input image size of 8×25, an output image size of 8×25, and 128 channels. C1_8 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 8×25, an output image size of 8×25, and a channel count of 1. SM1_4 is the activation layer, using softmax as the activation function. The input image size is 8×25, and the output image size is 8×25. C1_9 is a convolutional layer with a kernel size of 3×3, a stride of 1, an input image size of 8×25, an output image size of 8×25, and 256 channels. C1_10 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 8×25, an output image size of 8×25, and a channel count of 1. SM1_5 is the activation layer, using softmax as the activation function. The input image size is 8×25, and the output image size is 8×25. MP_1 is a pooling layer with a pooling filter size of 2×2, a stride of 2, an input image size of 16×50, and an output image size of 8×25. The convolutional layer C3_1, the adaptive average pooling layer AP3_1, and the activation layer RL3_1 are: The convolutional layer C3_1 has a kernel size of 3×3, a stride of 1, an input image size of 8×25, an output image size of 8×25, and 512 channels; the adaptive average pooling layer AP3_1 has an input image size of 8×25 and an output image size of 1×1; the activation layer RL3_1 uses Sigmoid as the activation function, has an input image size of 1×1, and an output image size of 1×1. The two convolutional layers and one activation layer are: The convolutional layer C2_1 has a kernel size of 3×3, a stride of 1, an input image size of 8×25, an output image size of 8×25, and a channel size of 496. The convolutional layer C2_2 has a kernel size of 1×1, a stride of 1, an input image size of 8×25, an output image size of 8×25, and a channel size of 1. The activation layer SM3_1 uses softmax as the activation function, with an input image size of 1×1 and an output image size of 1×1. The image under test is processed through a convolutional layer with 512 channels and a kernel size of 1×1 to obtain a first text feature. The first text feature is then processed by a max pooling layer with a pooling filter size of 2×2 to obtain a first text pooling feature, which is then input into the spatial location information perception module for processing. The first text feature is then processed through a convolutional layer with 256 channels and a kernel size of 1×1 to obtain a second text feature, which is then input into the spatial location information perception module for processing. The second text feature is then processed through a convolutional layer with 128 channels and a kernel size of 1×1 to obtain a third text feature, which is then input into the spatial location information perception module for processing. The third text feature is then processed through a convolutional layer with 64 channels and a kernel size of 1×1 to obtain a fourth text feature, which is then input into the spatial location information perception module for processing. The fourth text feature is then processed through a convolutional layer with 32 channels and a kernel size of 1×1 to obtain a fifth text feature, which is then input into the spatial location information perception module for processing. The first text pooling feature is processed by the C1_9 convolution to obtain the first text pooling convolution feature. The first text pooling convolution feature is then processed by the C1_10 convolution and then by the SM1_5 activation to obtain the first text pooling convolution processing feature. The first text pooling convolution processing feature and the first text pooling convolution feature are multiplied to obtain the first text multiplication feature. The first text multiplication feature and the first text pooling convolution feature are added to obtain the first text processing feature. The second text feature is processed by the C1_7 convolution to obtain the second text convolution feature. The second text convolution feature is then processed by the C1_8 convolution and then by the SM1_4 activation to obtain the second text convolution processed feature. The second text convolution processed feature and the second text convolution feature are multiplied to obtain the second text multiplied feature. The second text multiplied feature and the second text convolution feature are added to obtain the second text processed feature. The third text feature is processed by the C1_5 convolution to obtain the third text convolution feature. The third text convolution feature is then processed by the C1_6 convolution and then by the SM1_3 activation to obtain the third text convolution processing feature. The third text convolution processing feature and the third text convolution feature are multiplied to obtain the third text multiplication feature. The third text multiplication feature and the third text convolution feature are added to obtain the third text processing feature. The fourth text feature is processed by the C1_3 convolution to obtain the fourth text convolution feature. The fourth text convolution feature is then processed by the C1_4 convolution and then by the SM1_2 activation to obtain the fourth text convolution processed feature. The fourth text convolution processed feature and the fourth text convolution feature are multiplied to obtain the fourth text multiplied feature. The fourth text multiplied feature and the fourth text convolution feature are added to obtain the fourth text processed feature. The fifth text feature is processed by the C1_1 convolution to obtain the fifth text convolution feature. The fifth text convolution feature is then processed by the C1_2 convolution and then by the SM1_1 activation to obtain the fifth text convolution processed feature. The fifth text convolution processed feature and the fifth text convolution feature are multiplied to obtain the fifth text multiplied feature. The fifth text multiplied feature and the fifth text convolution feature are added to obtain the fifth text processed feature. The first text processing feature, the second text processing feature, the third text processing feature, the fourth text processing feature, and the fifth text processing feature are filtered and concatenated to obtain spatial location features; The first text pooling feature, the second text feature, the third text feature, the fourth text feature, and the fifth text feature are filtered and concatenated to obtain a global concatenated feature, which is then input into the global feature enhancement module for processing. The global concatenation feature is processed by the convolutional layer C2_1 to obtain the global convolutional feature. The global convolutional feature is then processed by the convolutional layer C2_2 and then by the activation layer SM3_1 to obtain the global processing feature. The global convolutional feature and the global processing feature are multiplied to obtain the global multiplied feature. The global multiplied feature and the global convolutional feature are added to obtain the global enhanced feature. The spatial location features and the global enhancement features are processed by the input feature fusion module, including: the spatial location features and the global enhancement features are processed by a convolutional layer C3_1 to obtain a preliminary fused feature; the preliminary fused feature is processed by an adaptive average pooling layer AP3_1 and then by an activation layer RL3_1 to obtain a preliminary fused processing feature; the preliminary fused processing feature and the preliminary fused feature are multiplied to obtain a fused multiplied feature; the fused multiplied feature and the preliminary fused feature are added to obtain a fused added feature; and the fused added feature is concatenated by a filter to obtain an output feature.

Images