Optimized scene text recognition system and method

Abstract

An optimization scene text recognition system and method, comprising: a text correction module, a Transformer feature extraction module and a feature decoding module, wherein: the text correction module corrects the curved text image in the input image, the Transformer feature extraction module extracts robust global features and initialization vectors from the corrected text image, and the feature decoding module guides the decoder to decode the extracted features to obtain the character sequence according to the initialization vector. In the feature extraction stage, the present application simply uses the Transformer encoder stack to extract strong and robust features and focus on global features, and after the feature extraction stage, content modeling is not required, and through the optimization of one-dimensional rectangular picture segmentation, the best precision is achieved on the public data set while reducing the amount of calculation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text recognition, and particularly relates to a scene text recognition system and method based on a Transformer and an initial vector guide, which can be used for text recognition of other language categories and various printed bodies, handwritten documents, card certificate texts, financial bills, license plates, education and the like. BACKGROUND

[0002] At present, in the field of computer vision, scene text recognition is a research hotspot. The purpose is to read out the text in the natural scene image. Since the recognized text contains a large amount of useful information, scene text recognition will play an important role in many subsequent sub-tasks, such as image search, key information extraction, visual question answering and the like. SUMMARY

[0003] The present application aims at the deficiency that the existing natural scene text recognition technology based on a Transformer model still needs a convolutional neural network in the feature extraction stage, and the initial vector in the decoding stage cannot be self-adaptive to different input pictures, and proposes an optimized scene text recognition system and method, which uses only a Transformer encoder stack to extract strong robust and global attention features in the feature extraction stage, does not need content modeling after the feature extraction stage, and optimizes one-dimensional rectangular picture segmentation, thereby reducing the amount of calculation while achieving the best accuracy on public data sets.

[0004] The present application is implemented by the following technical solutions:

[0005] The present application relates to a scene text recognition system based on a Transformer extractor and an initial vector guide, which comprises a text correction module, a Transformer feature extraction module and a feature decoding module, wherein: the text correction module corrects the curved text image in the input image, the Transformer feature extraction module extracts robust global features and an initial vector from the corrected text image, and the feature decoding module guides the decoder to decode the extracted features to obtain a character sequence according to the initial vector.

[0006] The Transformer feature extraction module comprises a slice encoding unit, an initial vector adding unit, a position encoding unit and a stacked Transformer encoder unit, wherein: the slice encoding unit converts the sliced image into a feature sequence; the initial vector adding unit adds the initial vector in front of the feature sequence, the position encoding unit adds position encoding to the feature sequence, and the stacked Transformer encoder unit extracts features from the feature sequence.

[0007] The slice is in the form of a one-dimensional rectangular image slice to reduce the amount of calculation.

[0008] The stacking refers to outputting a feature sequence of a previous Transformer encoder to a next Transformer encoder, and setting an attention residual unit between every two Transformer encoders to transmit attention.

[0009] The text correction module comprises a correction point acquisition unit and a TPS correction unit, wherein the correction point acquisition unit performs correction point prediction and obtains the position of the correction point of the text content according to the input image information, and the TPS correction unit performs TPS transformation processing on the image according to the position information of the correction point and obtains the result of the image correction.

[0010] The feature decoding module comprises a recurrent neural network, a character feature acquisition unit and a character prediction unit, wherein the recurrent neural network performs calculation and obtains the output result of the current frame according to the hidden state of the previous frame and the input information of the current frame, i.e., the character feature acquired by the character feature acquisition unit and the one-hot vector output by the character prediction unit at the previous moment, the character feature acquisition unit performs calculation and processing and obtains the character feature according to the hidden state information of the previous frame of the recurrent convolutional network and the feature extracted by the Transformer feature extraction module, and the character prediction unit performs softmax calculation and obtains the one-hot vector prediction character according to the output result of the recurrent neural network.

[0011] The output result is that, when in the first frame, the hidden state of the recurrent neural network is the initialization vector extracted in the Transformer feature extraction module.

[0012] The decoding refers to (x t , s t ) = RNN(s t-1 , (g t， , f(y t-1 ))) and p(y t ) = softmax(W0x t +b0), wherein s t is the hidden state at time t, g t is the character feature at time t, f(y t-1 ) is the one-hot vector output by the prediction at time t-1, s0 is the initialization vector extracted from the Transformer feature extraction module, x t is the output result of the recurrent neural network at time t, w0 and b0 are two learnable parameters, and then x t outputs the one-hot vector prediction character through the character prediction unit.

[0013] Technical effects

[0014] The present application solves the problems of the prior art that the feature decoding module needs to use a convolutional network and needs an additional module to assist feature extraction, and the decoding module cannot adaptively initialize the vector for different input pictures; through the adaptive initialization vector of the Transformer feature extraction module and the decoding module, one-dimensional rectangular segmentation is used for text images, and then the stacked Transformer encoder is used for feature extraction and decoding, which reduces the computational amount and has high recognition accuracy on commonly used data sets. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 It is a schematic diagram of the system of the present application.

[0016] Figure 2 It is a schematic diagram of the Transformer feature extraction module.

[0017] Figure 3 It is a schematic diagram of the feature decoding module. DETAILED DESCRIPTION

[0018] As shown in Figure 1 , it is a scene text recognition system based on Transformer and initialization vector guidance, which includes a text correction module, a Transformer feature extraction module and a feature decoding module, wherein: the text correction module corrects the curved text image in the input image, the Transformer feature extraction module extracts robust global features and initialization vectors from the corrected text image, and the feature decoding module guides the decoder to decode the extracted features to obtain the character sequence according to the initialization vector.

[0019] The present application is tested on seven public data sets IIIT5K, SVT, IC03, IC13, IC15, SVTP and CUTE data sets, and the accuracy reaches the best of the present on four of them.

[0020] The present application relates to a scene text recognition method of the above-mentioned system, which includes:

[0021] Step 1, training phase, including: generation of training data, construction of model, model training, wherein:

[0022] Step 1.1, select images from the training set, then enlarge the selected images to the same size according to a certain proportion.

[0023] Step 1.2: Construct the text correction module, Transformer feature extraction module, and feature decoding module sequentially. Predict control points using the CNN in the text correction module, and then use the Thin Plate Spline (TPS) interpolation algorithm to correct the curved image using the previously predicted control points. The corrected image is then output to the following... Figure 2 The Transformer feature extraction module shown above crops the image into rectangular slices and encodes them to obtain one-dimensional slice vectors. An initialization vector is added before several one-dimensional slice vectors to form a feature vector sequence. This sequence is then further encoded with positional codes and output to a stacked Transformer encoder to extract global features. i埨it f1, f2, ..., f 25 The first feature vector is then output to the feature decoding module. The feature decoding module initializes the recurrent neural network and character feature acquisition unit in the decoding module using the first feature vector as the initialization vector, and the remaining feature vectors serve as input to the character feature acquisition unit. The character feature acquisition unit obtains the features of the current character based on the hidden state and feature vectors of the recurrent neural network from the previous time step. The output of the recurrent neural network from the previous time step is combined with the one-hot encoding obtained by the character prediction unit and the character features output by the character feature acquisition unit, and then output to the recurrent neural network at the next time step. The features output by the recurrent neural network are then output to the character prediction unit to obtain the predicted character sequence.

[0024] The rectangular slicing method described here differs from the square slicing used in image classification tasks. For rectangular text, using rectangular slicing is more in line with the design of one-dimensional decoding, thereby reducing the amount of computation.

[0025] In the stacked Transformer encoders, an attention residual unit is provided between every two Transformer encoders to transfer the attention weights of the encoders.

[0026] The attention calculated by the attention residual unit Among them: Q i For query, K i For key, V i For value, P i The attention weights to be passed to the next Transformer encoder are i, where i is the Transformer encoder index and d is the dimension.

[0027] The embodiment uses 12 Transformer encoders, and the number of attention heads in each Transformer encoder is set to 16, and the dimension of each attention head is set to 32. The dimension of the full connection in the encoder is set to 2048.

[0028] The recurrent neural network in the feature decoder in the embodiment is GRU.

[0029] Step 1.3, the model constructed above is trained in a supervised training manner, wherein: the input of the model is a scene text image, and the output is a character sequence in the scene text image. The supervised loss used is cross-entropy loss: where y t is the annotated character of the scene text image. The optimizer used is Adadelta, and the learning rate changes with the training epoch. The learning rate is initially set to 1.0, and is reduced to 0.1 and 0.01 at the 19th and 23rd epochs, respectively. The entire model is trained for a total of 24 epochs.

[0030] During training, the random gradient descent algorithm is used to update the model parameters, and when the number of model iterations reaches the preset number of iterations, the model stops training. Every 2000 steps of training, a validation is performed to determine the accuracy. The optimal network model with higher accuracy is saved.

[0031] Step 2, scale the image to be tested and input the model constructed above to predict the character sequence contained in the image. When decoding the features, the beam search method is used to search for characters, and the number of candidates is set to 5 at this time.

[0032] The embodiment uses the trained scene text recognition method based on Transformer and initial vector guidance to test on seven commonly used scene text recognition data sets. The seven commonly used scene text recognition data sets include: IIIT5K, SVT, IC03, IC13, IC15, SVTP and CUTE.

[0033] 1) Introduction of the training data set: The optimized scene recognition system of the embodiment is trained on two publicly available generated training sets: MJSynth and SynText. The network model of the embodiment is not tuned for a separate test set, but only uses IIIT as a validation set to search for the optimal network model.

[0034] 2) In the embodiment, all images are scaled to 64*256, randomly shuffled and combined into a batch to output to the model constructed above.

[0035] 3) After the image is processed by the text correction module, a shallow CNN network predicts 20 control points. TPS uses these 20 control points to correct the image to a 32*100 size image. After that, the image is cut into a 32*4 size rectangle slice, and then converted into a feature vector sequence with a length of 25 and a dimension of 384. After that, the feature vector sequence is mapped to a feature vector sequence with a length of 25 and a dimension of 512 through a fully connected layer. After adding an initial vector to the feature vector sequence, a feature vector sequence with a length of 26 and a dimension of 512 is formed. Finally, the word feature vector sequence is added with position encoding and output to the Transformer feature extraction module.

[0036] The method of the embodiment achieves the best results on four of the seven public data sets (SVT, IC13, IC15 and SVTP) as shown in Table 2. Good results are also achieved on other data sets.

[0037] Table 1 Performance comparison of different text recognition methods on seven public data sets (for fairness, only methods that also use MJ and ST as training sets are compared)

[0038]

[0039] As shown in Table 2, experiments show that using the initial vector obtained from the Transformer feature extraction module can better guide the decoder to obtain higher accuracy.

[0040] Table 2 Comparison with and without initial vector guidance

[0041] Method Average accuracy on seven datasets With initialization vector guidance 90.8 Without initialization vector guidance 90.6

[0042] Table 3 proves that the method of the embodiment does not need to additionally add a content modeling module, and the Transformer feature extraction module in the method can extract strong and robust features.

[0043] Table 3 Accuracy comparison with and without BiLSTM content modeling

[0044]

[0045] Compared with the prior art, the method extracts more robust features through the Transformer feature extraction and focuses on the global and adaptive initial vector guidance in the decoding module for different input pictures.

[0046] The above specific embodiments can be partially adjusted in different ways by those skilled in the art without departing from the principles and purposes of the present application, the protection scope of the present application is subject to the claims and is not limited by the above specific embodiments, each implementation within the scope is subject to the present application.

Claims

1. A scene text recognition system based on Transformer extractor and initial vector guidance, characterized in that, include: The system comprises a text correction module, a Transformer feature extraction module, and a feature decoding module. The text correction module corrects curved text images in the input image. The Transformer feature extraction module extracts robust global features and initialization vectors from the corrected text image. The feature decoding module guides the decoder to decode the extracted features based on the initialization vectors to obtain the character sequence. In the feature extraction stage, a stack of Transformer encoders is used to extract robust and globally relevant features. No content modeling is required after the feature extraction stage. Furthermore, by optimizing one-dimensional rectangular image segmentation, the system achieves optimal accuracy on public datasets while reducing computational cost. The Transformer feature extraction module includes: a slice encoding unit, an initialization vector addition unit, a position encoding unit, and stacked Transformer encoder units, wherein: the slice encoding unit converts the sliced ​​image into a feature sequence; the initialization vector addition unit adds an initialization vector before the feature sequence; the position encoding unit adds position encoding to the feature sequence; and the stacked Transformer encoder units extract features from the feature sequence.

2. The scene text recognition system based on Transformer extractor and initial vector guidance according to claim 1, characterized in that, The stacking refers to the following: the feature sequence output by the previous Transformer encoder is output to the next Transformer encoder, and an attention residual unit is provided between every two Transformer encoders to transfer attention.

3. The scene text recognition system based on Transformer extractor and initial vector guidance according to claim 2, characterized in that, The attention calculated by the attention residual unit ,in: For query, For key, For value, , The attention weights to be passed to the next Transformer encoder are i, where i is the Transformer encoder index and d is the dimension.

4. The scene text recognition system based on Transformer extractor and initial vector guidance according to claim 1, characterized in that, The text correction module includes a correction point acquisition unit and a TPS correction unit. The correction point acquisition unit predicts correction points based on the input image information and obtains the positions of the correction points in the text content. The TPS correction unit performs TPS transformation on the image based on the position information of the correction points and obtains the image correction result.

5. The scene text recognition system based on Transformer extractor and initial vector guidance according to claim 1, characterized in that, The feature decoding module includes a recurrent neural network, a character feature acquisition unit, and a character prediction unit. The recurrent neural network calculates and obtains the output result of the current frame based on the hidden state of the previous frame and the input information of the current frame, namely the character features acquired by the character feature acquisition unit and the one-hot vector output by the character prediction unit at the previous time step. The character feature acquisition unit calculates and obtains the character features based on the hidden state information of the previous frame of the recurrent convolutional network and the features extracted by the Transformer feature extraction module. The character prediction unit performs softmax calculation based on the output result of the recurrent neural network and obtains the one-hot vector to predict the character.

6. The scene text recognition system based on Transformer extractor and initial vector guidance according to claim 5, characterized in that, The output result, in the first frame, is the hidden state of the recurrent neural network, which is the initialization vector extracted by the Transformer feature extraction module.

7. The scene text recognition system based on Transformer extractor and initial vector guidance according to claim 1 or 5, characterized in that, The decoding mentioned above refers to: , ,in: Let be the hidden state at time t. The character features at time t, Let be the one-hot vector of the predicted output at time t-1. The initialization vector is extracted from the Transformer feature extraction module. This represents the output of the recurrent neural network at time t. These are two learnable parameters, then... It will use the character prediction unit to output a one-hot vector to predict characters.

8. The scene text recognition method according to any one of claims 1 to 7, characterized in that, include: Step 1, the training phase, specifically includes: Step 1.1: Select images from the training set, and then scale the selected images to the same size; Step 1.2: Construct the text correction module, Transformer feature extraction module, and feature decoding module sequentially. The text correction module uses a CNN to predict control points, and then uses a thin-plate spline interpolation algorithm to correct the curved image using the previously predicted control points. The corrected image is then output to the Transformer feature extraction module. The Transformer feature extraction module crops the image into rectangular slices and encodes them to obtain one-dimensional slice vectors. An initialization vector is added before several one-dimensional slice vectors to form a feature vector sequence. Position encoding is then added before outputting the sequence to a stacked Transformer encoder to extract global features. The output is then sent to the feature decoding module. The feature decoding module uses the first feature vector as the initialization vector to initialize the recurrent neural network and character feature acquisition unit in the decoding module, and the remaining feature vectors are used as inputs to the character feature acquisition unit. The character feature acquisition unit obtains the features of the current character based on the hidden state of the recurrent neural network in the previous time step and the feature vector. The output of the recurrent neural network in the previous time step is combined with the one-hot encoding obtained by the character prediction unit and the character features output by the character feature acquisition unit, and then output to the recurrent neural network in the next time step. The features output by the recurrent neural network are then output to the character prediction unit to obtain the predicted character sequence. Step 1.3: Train the constructed model using supervised training, where the model's input is a scene text image, and the output is a sequence of characters from the scene text image; the supervised loss used is cross-entropy loss. ,in These are characters used for scene text image annotations; Step 2: Scale the image to be tested and input the character sequence contained in the image predicted by the model constructed above.

Images