A scene text recognition system based on consistent regularization training

Abstract

The present invention relates to a scene text recognition system based on consistency regularization training, and belongs to the field of scene text recognition. The whole system includes three branches, including a supervised branch, an unsupervised branch and a domain adaptation branch. The present invention uses an application consistency regularization method to train a more robust and better-performing STR model. Specifically, the STR model receives two enhanced views of unlabeled text images as input and forces it to output the same result. Through this training method, the model can utilize large-scale unlabeled data to learn transformation invariant features. The present invention adds a projection module in a path of the unsupervised branch to prevent the solution of model collapse. Considering the huge domain gap between training data and real test data, domain adaptation loss is applied to narrow the distance between character-level features between synthetic labeled data and real unlabeled data.

Description

Technical Field

[0001] The present invention belongs to the field of scene text recognition, and in particular is a scene text recognition system based on consistency regularization training. Background Art

[0002] Scene text recognition (STR) is the recognition of text in natural scenes and is a special form of optical character recognition (OCR).

[0003] In the past, handcrafted features were used for scene text recognition, such as histogram of oriented gradient descriptors, connected components, and stroke width transforms. With the rapid development of deep learning technology, scene text recognition has made great progress in terms of innovation, practicality, and efficiency. There are two main categories of scene text recognition, namely, segmentation-based methods and methods without segmentation. In particular, methods without segmentation can be roughly divided into methods based on connection temporal classification (CTC) and methods based on attention.

[0004] At present, regular text recognition methods have achieved good performance due to the application of convolutional neural networks and attention mechanisms. Compared with the regular text recognition task, irregular text recognition is more difficult due to the multiple interferences of the environment, various shapes and distorted patterns.

[0005] Although deep learning methods have achieved great success in many computer vision tasks including scene text recognition, it has a great demand for big data. In addition to real labeled data, synthetic labeled data is also widely used to train STR models. Synthetic data and real data each have their own advantages and disadvantages. Real data is manually annotated, but it is usually expensive, time-consuming and small in size. Synthetic data is automatic and efficient, but it is difficult to design a good synthetic data engine for different tasks, and there is always a domain gap between synthetic data and real data.

[0006] Considering that it is easier to collect unlabeled data in the real world, many researchers intend to use unlabeled data to improve the performance of deep models. Semi-supervised methods can combine additional unlabeled data with labeled data during training, and the most commonly used is self-training methods. Compared with self-training methods such as pseudo-labeling, consistency regularization methods such as UDA (Unsupervised Data Augmentation) are generally considered to be more effective and efficient. It is well known that consistency regularization methods have not been successfully applied to scene text recognition tasks. When the consistency regularization method was applied to the STR model, it was found that the model collapsed severely and the performance was poor. Summary of the invention

[0007] Technical issues to be solved

[0008] In order to solve the above problems, the present invention proposes an asymmetric unsupervised branch structure and combines it with a domain adaptation branch, which effectively improves the model training stability and final performance.

[0009] Technical Solution

[0010] A scene text recognition system based on consistency regularization training, characterized by comprising a supervised branch, an unsupervised branch and a domain adaptation branch; the supervised branch receives a text image X L As input, and calculate the predictive distribution P of the student model L and the annotation text string Y gt The cross entropy loss between the unlabeled image X and the unsupervised branch is enhanced by weak data enhancement and strong data enhancement. U Convert to two enhanced views X respectively U1 and X U2 ; For the input image X U1 , the teacher model outputs the predicted distribution P U1 ; For the input image X U2 , the student model outputs the predicted distribution P U2 ; Then, the weak data is enhanced to view X U1 The predicted distribution P U1 As the goal, we use a consistency regularization loss to force a strong data augmentation view X U2 The output P U2 Closer to P U1 ; The domain adaptation branch calculates the CORAL loss to align the character-level features of labeled data and unlabeled data; Finally, the weighted sum of the above three losses is calculated, and the model is updated using the back-propagation algorithm.

[0011] The student model and the teacher model have the same structure but different parameters, and both use a scene text recognition model.

[0012] The scene text recognition model includes a correction module, a feature extraction module, a sequence modeling module and a prediction module.

[0013] A projection module is added to the student model before the classification layer in the unsupervised branch.

[0014] The projection module is a feed-forward layer module PFF.

[0015] Beneficial Effects

[0016] The present invention proposes a scene text recognition system based on consistency regularized training. The whole system includes three branches, including supervised training using labeled data, consistent regularized training using unlabeled data, and domain adaptation branches for closing the domain distance between synthetic data and real data. The present invention uses the application consistency regularization method to train a more robust and better-performing STR model. Specifically, the STR model receives two enhanced views of unlabeled text images as input and forces it to output the same result. Through this training method, the model can use large-scale unlabeled data to learn transformation invariant features. The present invention adds a projection module in a path of the unsupervised branch to prevent the solution of model collapse. Considering the huge domain gap between training data and real test data, domain adaptation loss is applied to close the distance between character-level features between synthetic labeled data and real unlabeled data.

[0017] The present invention can effectively use real unlabeled data to improve the robustness and performance of the STR model. Compared with the supervised baseline, the semi-supervised model of the present invention shows great improvement. Without using manually annotated data, the present invention can outperform the current state-of-the-art methods in all scene text recognition test datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0019] Figure 1 System overall structure diagram;

[0020] Figure 2 TRBA flow chart;

[0021] Figure 3 Diagram of the domain adaptation branch structure. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0023] like Figure 1 As shown in FIG. 1 , the system of the present invention consists of three branches, namely the supervised branch, the unsupervised branch and the domain adaptation branch. It has two scene text recognition models, namely the student model and the teacher model. The student model and the teacher model have the same structure but different parameters. The supervised branch accepts the text image X LAs input, and calculate the predictive distribution P of the student model L and the annotation text string Y gt The unsupervised branch transforms the unlabeled image X into U Convert to two enhanced views X respectively U1 and X U2 For an input image X U1 , the teacher model outputs the predicted distribution P U1 For an input image X U2 , the student model outputs the predicted distribution P U2 Then, the weak data enhancement view X U1 The predicted distribution P U1 As the goal, we use a consistency regularization loss to force a strong data augmentation view X U2 The output P U2 Closer to P U1 In order to narrow the domain gap between synthetic labeled data and real labeled data, the present invention calculates the CORAL loss to align the character-level features of labeled data and unlabeled data. Finally, the weighted sum of the above three losses is calculated, and the model is updated using the back-propagation algorithm.

[0024] Scene text recognition model. The scene text recognition model consists of a correction module (specifically thin plate spline TPS), a feature extraction module (convolutional residual network ResNet), a sequence modeling module (bidirectional BiLSTM) and a prediction module (attention mechanism Attention), such as Figure 2 As shown, it is called TRBA. The rectification module uses thin plate spline (TPS) to transform the input image X into a normalized image X, which is used to reduce the difficulty of subsequent recognition tasks. For an input image size of 32×100, the feature map size of the last convolutional layer of the feature extraction module is C×1×T (C is the channel size and T is the maximum sequence length of the decoder). The sequence modeling module uses a bidirectional long short-term memory recurrent neural network (BiLSTM) to learn better sequence features H∈T×C. The prediction module uses attention-based sequence prediction. At the tth time step of decoding, the prediction module uses the encoder output H, the last hidden state, and the character s t-1 And the embedding information f(y t -1) to predict the character to predict the output of the current time step.

[0025] e t,i =w t tanh(Ws t-1 +Vh i +b)

[0026]

[0027] Where w, W, V are trainable parameters. The attention weight α represents the importance of the sequence feature H at different time steps. With the weight as the coefficient, the decoder weights H as the feature vector g t , which is treated as a character-level feature in the domain adaptation branch. The recurrent unit of the prediction module converts g t Takes visual input and produces an output vector s t .

[0028]

[0029] s t =rnn(s t-1 ,(g t ,f(y t-1 )))

[0030] After the classifier layer, the TRBA model outputs a sequence prediction result of size T×K, where K is the number of character categories.

[0031] The supervised branch will label the image X in the data. L As input, the output P is obtained after the correction, feature extraction, sequence modeling and prediction stages of the student model. L . Calculate the prediction results of the student model and the labeled text string Y gt The cross entropy loss between:

[0032]

[0033] where θ student are the model parameters of the student model.

[0034] Unsupervised branch. The unsupervised branch only requires unlabeled text images X U , no corresponding annotation string is required. According to the different strengths of the data enhancement strategy, the present invention predefines two different data enhancement strategies, namely weak (data) enhancement and strong (data) enhancement. U Transformed to X through weak enhancement U1 , transformed into X through strong enhancement U2 .

[0035]

[0036]

[0037]

[0038] Where I is the indicator function. In practice, if the student model and the teacher model share the same model structure and parameters, the unsupervised branch will collapse severely and perform poorly. Therefore, the present invention adds a projection module to the student model before the classification layer to destroy this symmetric structure. The present invention uses the position-wise feedforward (PFF) module proposed in the Transformer as the projection module. In addition, the present invention uses the exponential moving average method to update the teacher model to keep the projection module in the best state.

[0039] Domain adaptation branch. Figure 3 As shown in Figure 1, the present invention uses domain adaptation loss to narrow the domain gap between synthetic labeled data and real unlabeled data. Instead of aligning the global feature space between synthetic data and real data, the present invention adaptively focuses on aligning the distribution of synthetic data and real data in the character-level feature space. The present invention regards the vector g that automatically focuses on local features in each decoding time step as a character-level feature. The correlation loss (CORrelation Alignment, CORAL) is used to measure F L clf and F U2 clf The distribution distance between two character-level features. The domain adaptation loss is defined as follows:

[0040]

[0041] Among them U L Yes F L clf The collection of U U2 Yes F U2 clf A collection of represents the Frobenius norm of the square matrix, and cov(U) is the covariance matrix.

[0042] Training process and inference process. During the training process, the final loss is defined as follows:

[0043] L=L sup +λ unl L unl +λ da L da

[0044] Where L unl and L da is a hyperparameter. Adam is selected as the optimizer and back-propagation is used to update the parameters of the student model. The gradient of the teacher model is stopped and its parameters are updated using the exponential moving average (EMA) mechanism, which is defined as follows:

[0045] θteac =αθ teacher +(1-α)θ student

[0046] Where α is a hyperparameter. After training, only the student model is saved and used to predict the content of the text image in the inference phase. In the inference phase, the prediction module first uses the "BOS" character representing the start of decoding as input and outputs the first character. Next, the prediction module iteratively uses the previous output character as input in each decoding time step of the inference phase. Once the "EOS" character is output, the prediction module ends.

[0047] The following will introduce the training and test data sets and the experimental settings of the system used in the present invention.

[0048] (1) Dataset

[0049] We train our system on two synthetic labeled datasets, Synth90K and SynthText, with a total of about 14.5M samples. We use about 10.7M real unlabeled data. We evaluate our system on six scene text recognition benchmark test datasets. Both the synthetic labeled data and the test datasets are public, while the real unlabeled data is private.

[0050] The details of the six test datasets are as follows. IIT5K-Words (IT5K) contains 3000 cropped word images for testing. Street View Text (SVT) consists of 647 word images, which are collected from Google Street View. Many images are of low resolution or have a lot of noise and blur. ICDAR 2013 (IC13) contains 1015 cropped word images, and IC13_857 is a subset of IC13 without word images shorter than 3 characters. SVT-Perspective (SVT-P) contains 645 cropped images for testing. Most of the images are perspective-distorted because the images are selected from side-view snapshots in Google Street View. CUTE80 contains 288 high-resolution images for testing, but some images are curved. ICDAR 2015 (IC15 contains 2077 cropped word images. IC15_1811 is a subset of IC15, discarding word images with non-alphanumeric characters.

[0051] The details of the real unlabeled datasets are as follows. All unlabeled datasets contain about 10.7 billion word images cropped from three scene image datasets, including Places2, OpenImages, and ImageNet ILSVRC 2012. The present invention uses a scene text detector named CRAFT (Character-Region Awareness For Text detection) to detect and crop text images. The text confidence threshold is set to 0.7, and low-resolution (width times height less than 1000) text images are discarded. No other post-processing methods are applied, and finally 10.7M text images are obtained.

[0052] (2) Data enhancement

[0053] Data augmentation plays an important role in the consistency regularization training of the present invention. As mentioned above, weak (data) augmentation and strong (data) augmentation are used for unlabeled images in the unsupervised branch. In order to facilitate fair comparison with fully supervised training, strong augmentation is also used in the training of the supervised branch. The weak augmentation method changes the brightness, contrast, saturation and hue of the image. The strong augmentation method is inherited from the RandAugment augmentation method. It includes channel transformations such as auto-contrast, contrast, brightness, color, balance, color separation, exposure, exposure addition operation space transformations such as shearing, translation and rotation, etc.

[0054] (3) Experimental setup

[0055] The models and systems of the present invention are implemented using PyTorch.

[0056] In the preprocessing stage, the original images are converted into enhanced images through different enhancement strategies and resized to the same size of 32 × 100. In order to balance the ratio of labeled data and unlabeled data, the batch size N of labeled data is set. L The batch N is 384 and the unlabeled data is U Up to batches of 128.

[0057] The present invention uses the Adam optimizer to train the system. In order to accelerate the convergence of the model, the OneCycleLR learning rate scheduler is used, and the maximum learning rate is set to 0.001. The OneCycleLR learning rate scheduler first increases the learning rate from the base learning rate to the maximum learning rate, and then gradually decreases it until it drops to the base learning rate at the end of training. The temperature coefficient τ is set to 0.4 and the confidence threshold β is 0.3. The exponential moving average decay ratio is set to 0.999 to keep the projection layer in the best state. The present invention sets λunl and λda to 1.0 and 0.01 respectively. The system combines labeled data and unlabeled data training for a total of 250,000 iterations.

[0058] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should be included in the protection scope of the present invention.

Claims

1. A scene text recognition system based on consistency regularization training, characterized by It includes a supervised branch, an unsupervised branch and a domain adaptation branch; the supervised branch accepts a text image X L As input, and calculate the predicted distribution P of the student model L and the annotation text string Y gt The cross entropy loss between the unlabeled images X U Convert to Enhanced View X U1 , through strong data augmentation, the unlabeled image X U Convert to Enhanced View X U2 ; For the input image X U1 , the teacher model outputs the predicted distribution P U1 ; For the input image X U2 , the student model outputs the predicted distribution P U2 ; Then, the weak data is enhanced to view X U1 The predicted distribution P U1 As the goal, we use a consistency regularization loss to force a strong data augmentation view X U2 The output P U2 Closer to P U1 ; The domain adaptation branch calculates the CORAL loss to align the character-level features of the labeled data and the unlabeled data; Finally, the weighted sum of the above three losses is calculated and the model is updated using the back-propagation algorithm; A projection module is added to one path of the unsupervised branch to prevent the model from collapsing; The supervised branch accepts a text image X L As input, and calculate the predicted distribution P of the student model L and the annotation text string Y gt The cross entropy loss between: Where T is the maximum sequence length of the decoder, θ student are the model parameters of the student model; The predicted distribution P U1 and the predicted distribution P U2 The calculation formula is: Among them, β is the confidence threshold, KL is the divergence function, and I indicates whether it is greater than the threshold. If it is, the output is 1, otherwise it is 0; The domain adaptation branch calculates the CORAL loss to align the character-level features of the labeled data and the unlabeled data, specifically: The vector g that automatically focuses on local features at each decoding time step is regarded as a character-level feature, and the correlation loss CORAL is used to measure F L clf and F U2 clf The distribution distance between two character-level features; the domain adaptation loss is defined as follows: Among them, U L Yes F L clf The collection of U U2 Yes F U2 clf A collection of represents the Frobenius norm of the square matrix, and cov(U) is the covariance matrix.

2. A scene text recognition system based on consistency regularization training according to claim 1, characterized in that The student model and the teacher model have the same structure but different parameters, and both use a scene text recognition model.

3. A scene text recognition system based on consistency regularization training according to claim 2, characterized in that The scene text recognition model includes a correction module, a feature extraction module, a sequence modeling module and a prediction module.

4. A scene text recognition system based on consistency regularization training according to claim 1, characterized in that A projection module is added to the student model before the classification layer in the unsupervised branch.

5. A scene text recognition system based on consistency regularization training according to claim 4, characterized in that The projection module is a feed-forward layer module PFF.

Images