A Bidirectional Scene Text Recognition Method Based on Transformer
By introducing cross-gated fusion mechanism and direction embedding into the Transformer structure, combining image-level and character-level features, the problem of lack of scene context and first character recognition error rate in the prior art is solved, and the accuracy of scene text recognition is significantly improved sex.
Patent Information
- Application Number
- CN202411207610.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-30
AI Technical Summary
Existing text recognition methods based on cropped images cannot effectively utilize global scene information in the case of blurred graphics, low resolution, and corruption, resulting in poor accuracy of scene text recognition, especially high error rate of first character recognition.
Using a two-way scene text recognition method based on Transformer, the BiSTR network is constructed, including image encoder, ViT character encoder, cross-gated fusion mechanism and bidirectional decoder, and the multi-head cross-attention mechanism and hyperbolic tangent-gated mechanism will be used to build a BiSTR network, including image encoder, ViT character encoder, cross-gated fusion mechanism, and bidirectional decoder. Image-level and character-level features are merged, and direction embedding is introduced in the Transformer structure to perform bidirectional decoding of context information.
It improves the model's ability to obtain full-text information, improves the accuracy of scene text recognition, and significantly reduces the first character recognition error rate.
Smart Images

Figure CN118711189B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image recognition, and in particular to a Transformer-based bidirectional scene text recognition method. Background Art
[0002] In recent years, research on scene text recognition has made significant breakthroughs. Scene text recognition is divided into two tasks: text detection and recognition. Most recognition research is implemented on the detected text area. In cases of blurred, low-resolution, damaged graphics, etc., global scene information is very important for text recognition, while text recognition methods based on cropped images cannot provide this information.
[0003] Existing technologies combine convolutional neural networks and recurrent neural networks to recognize scene text. The deep learning-based method uses CNN to extract the visual features of the image, and then uses RNN to learn the contextual features of the image. RNN learns the attention between sequences, but there is a problem of poor parallelism that affects efficiency. Summary of the invention
[0004] Purpose of the invention: The purpose of the present invention is to provide a Transformer-based bidirectional scene text recognition method, which solves the problem of lack of scene context in cropping-based text recognizers and the problem of high error rate in first character recognition.
[0005] Technical solution: The present invention discloses a Transformer-based bidirectional scene text recognition method, comprising the following steps:
[0006] (1) Obtain the dataset from the open source website and divide it into training set and test set;
[0007] (2) Construct and train the BiSTR bidirectional scene text recognition network, including: image encoder module, ViT character encoder module, cross-gated fusion mechanism module, and Transformer-based bidirectional decoder module;
[0008] (3) Obtain the final prediction result.
[0009] Furthermore, step (1) is specifically as follows: for the training dataset, a synthetic text dataset is selected to train the network; the network performance is evaluated on seven common test sets, including the handwritten text dataset IIIT, the scene text data IC13, the scene text data C03, the scene text data IC15, the street view text dataset SVT, the street view text-perspective dataset SVT-P, and the curved text dataset CUTE.
[0010] Furthermore, the image encoder module is used to input the image into the ResNet image encoder to extract image-level features; the character encoder module is used to input the image into the ViT character encoder to extract character-level features.
[0011] Furthermore, the cross-gated fusion mechanism module is used to integrate image-level and word-level features; it consists of a multi-head cross-attention mechanism and a hyperbolic tangent-gating mechanism; the global features are linearly projected to the local representation, and the local representation is dimension d; the global features are regarded as visual content, and the global features are merged with the cropped visual features using the multi-head cross-attention mechanism, and the formula is as follows:
[0012] ;
[0013] Among them, Q represents the query vector, K represents the key vector, and V represents the value vector. represents image-level features, Represents character-level features;
[0014] A hyperbolic tangent-gating mechanism is added to the multi-head cross attention mechanism. The hyperbolic tangent-gating mechanism is used to maintain the integrity of the forward pass at initialization and gradually transform between the original global features and the fused representation during training. The formula is as follows:
[0015] ;
[0016] Among them, tanh() is the hyperbolic tangent function, is the character level feature, is a learnable parameter, initialized to 0; Represents the features of the multi-head attention mechanism output; Represents the features of the gated fusion output.
[0017] Furthermore, the Transformer-based bidirectional decoder module is as follows: the result obtained by the fusion mechanism is input into the Transformer bidirectional decoder after feature combination. The Transformer bidirectional decoder adds direction embedding on the basis of position embedding and tag embedding, and decodes in original and reverse character order. The model generates two predictions for a given sample, and uses cross entropy loss to calculate the probability of each type of true value. Finally, the output with the highest word probability will be selected as the final prediction.
[0018] Furthermore, the same decoder network is used to condition the reading direction of the output sequence by adding a directional embedding, where each input image in the batch is decoded twice, from left to right and from right to left, in each training iteration.
[0019] Furthermore, we unbind the position encoding strategy in the input token embedding, calculate the position attention, and then add the position attention in the multi-head cross attention mechanism; the position attention formula is as follows:
[0020] ;
[0021] ;
[0022] Among them, j represents the truth value type in different directions, represents the position embedding of the encoder part, represents the position code, , , , is a linear layer, d represents the dimension of the linear layer, and all layers in the decoder share the same position attention; and Both indicate position attention.
[0023] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: a cross-gated fusion mechanism is proposed to merge image-level features and character-level features to improve the model's ability to obtain full-text information; directional embedding is introduced into the Transformer structure to provide decoding in two different directions, and the context information complements each other, thereby improving the accuracy of the scene text recognition model. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 is a flow chart of the present invention;
[0025] Figure 2 is a network structure diagram of the present invention;
[0026] Figure 3 It is a structural diagram of the cross-attention mechanism of the present invention;
[0027] Figure 4 This is a network structure diagram of the bidirectional decoder of the present invention. DETAILED DESCRIPTION
[0028] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.
[0029] like Figure 1 As shown, an embodiment of the present invention provides a Transformer-based bidirectional scene text recognition method, comprising the following steps:
[0030] (1) Data sets were obtained from open source websites and divided into training sets and test sets. The details are as follows: For the training data set, a synthetic text data set was selected to train the network; the network performance was evaluated on seven common test sets, including the handwritten text data set IIIT, the scene text data set IC13, the scene text data set C03, the scene text data set IC15, the street view text data set SVT, the street view text-perspective data set SVT-P, and the curved text data set CUTE.
[0031] (2) If Figure 2 As shown in the figure, a BiSTR bidirectional scene text recognition network is constructed and trained, including: an image encoder module, a ViT character encoder module, a cross-gated fusion mechanism module, and a Transformer-based bidirectional decoder module; wherein, the image encoder module is used to input the image into the ResNet image encoder to extract image-level features; specifically as follows: ResNet mainly includes an input layer, and the image data is fed into the first convolutional layer of ResNet (usually a 7×7 convolutional layer) as input for preliminary extraction of image features; the residual block image data is further extracted through a series of residual blocks (each residual block contains two convolutional layers); downsampling and fully connected layers and outputs. Throughout the entire workflow, batch normalization (Batch Norm) and ReLU activation functions are usually used after each convolutional layer and fully connected layer to accelerate training and improve the generalization ability of the model. The size of the output feature of the image encoder ResNet is , k represents the kernel of average pooling.
[0032] The character encoder module is used to input the image into the ViT character encoder to extract character-level features. Specifically: the input cropped visual image is encoded; ViT divides the image into patches, and then the patches are converted into one-dimensional vector embeddings. A learnable patch embedding is added together with the positional encoding of each embedding as the input of the encoder.
[0033] Input Image is reshaped into a sequence of flat 2D patches The image dimension is , the number of channels is C, and the dimension of the patch is , the resulting patch sequence is of length N; the Transformer encoder uses constant width d to embed and feature in all its layers. To match this size, each tiled patch is converted to an embedding of size d by linear projection, and the sequence is prepended to a learnable class embedding of the same dimension d, adding a unique positional encoding of the same dimension d to each embedding, and the resulting vector sum is the input to the encoder.
[0034] like Figure 3As shown in the figure, the cross-gated fusion mechanism module is used to integrate image-level and word-level features; it consists of a multi-head cross-attention mechanism and a hyperbolic tangent-gating mechanism; the global features are linearly projected to the local representation, and the local representation is dimension d; the global features are regarded as visual content, and the multi-head cross-attention mechanism is used to merge the global features with the cropped visual features. The formula is as follows:
[0035] ;
[0036] Among them, Q represents the query vector, K represents the key vector, and V represents the value vector. represents image-level features, Represents character-level features;
[0037] A hyperbolic tangent-gating mechanism is added to the multi-head cross attention mechanism. The hyperbolic tangent-gating mechanism is used to maintain the integrity of the forward pass at initialization and gradually transform between the original global features and the fused representation during training. The formula is as follows:
[0038] ;
[0039] Among them, tanh() is the hyperbolic tangent function, is the character level feature, is a learnable parameter, initialized to 0; Represents the features of the multi-head attention mechanism output; Represents the features of the gated fusion output.
[0040] in, Figure 3 The parameters are as follows: In the cross attention structure diagram As the input of the encoder, we first multiply the transformation matrix to get the embedding ( , , ), embedding is multiplied by the transformation matrix, and each position generates a set of key vectors ( , , ) and the value vector ( , , ). As the input of the encoder, a query vector (q) is generated, which is used to calculate the attention weights at all positions of the encoder. The attention weights are obtained by calculating the similarity between the query vector and the key vector and normalizing them with the softmax function. Finally, the attention weights are multiplied and summed with the value vector to obtain the adjusted output of the encoder for use by the decoder.
[0041] like Figure 4As shown in the figure, the Transformer-based bidirectional decoder module is as follows: the result obtained by the fusion mechanism is input into the Transformer bidirectional decoder after feature combination. The Transformer bidirectional decoder adds direction embedding on the basis of position embedding and tag embedding, and decodes in original and reverse character order at the same time. The model generates two predictions for a given sample, and uses cross entropy loss to calculate the probability of each type of true value. Finally, the output with the highest word probability will be selected as the final prediction. In the training iteration, each input image in the batch is decoded twice from left to right and from right to left.
[0042] Unbind the position encoding strategy in the input token embedding, calculate the position attention, and then add the position attention in the multi-head cross attention mechanism; the position attention formula is as follows:
[0043] ;
[0044] ;
[0045] Among them, j represents the truth value type in different directions, represents the position embedding of the encoder part, represents the position code, , , , is a linear layer, d represents the dimension of the linear layer, and all layers in the decoder share the same position attention; and Both indicate position attention.
[0046] The specific training parameters are as follows: The Transformer-based encoder and decoder both contain 6 encoder layers and 6 decoder layers. The encoding dimension is 512, and the multi-head attention is 16 heads. The intermediate size of the feedforward layer is 2048. The model recognizes 100 classes for training, including 10 numbers, 52 case-sensitive letters, 35 punctuation marks, a start token, an end token, and a pad token. All experiments were trained for 10 cycles. The images were grayscaled and resized to a height and width of 32 × 128 without retaining the original aspect ratio.
[0047] (3) Obtain the final prediction result.
Claims
1. A bidirectional scene text recognition method based on Transformer, characterized in that: The following steps are involved: (1) Obtain the dataset from the open source website and divide it into training set and test set; (2) Construct a BiSTR bidirectional scene text recognition network and train it, including: an image encoder module, a ViT character encoder module, a cross-gated fusion mechanism module, and a Transformer-based bidirectional decoder module; the image encoder module is used to input the image into the ResNet image encoder to extract image-level features; the ViT character encoder module is used to input the image into the ViT character encoder to extract character-level features; the cross-gated fusion mechanism module is used to integrate image-level and word-level features; it consists of a multi-head cross-attention mechanism and a hyperbolic tangent-gating mechanism; the global features are linearly projected to the local representation; the global features are regarded as visual content, and the global features are merged with the cropped visual features using the multi-head cross-attention mechanism, as shown in the following formula: ; Among them, Q represents the query vector, K represents the key vector, and V represents the value vector. represents image-level features, Represents character-level features; A hyperbolic tangent-gating mechanism is added to the multi-head cross attention mechanism. The hyperbolic tangent-gating mechanism is used to maintain the integrity of the forward pass at initialization and gradually transform between the original global features and the fused representation during training. The formula is as follows: ; Among them, tanh() is the hyperbolic tangent function, is the character level feature, is a learnable parameter, initialized to 0; Represents the features of the multi-head attention mechanism output; Represents the features of the gated fusion output; The Transformer-based bidirectional decoder module is as follows: the result obtained by the fusion mechanism is input into the Transformer bidirectional decoder after feature combination. The Transformer bidirectional decoder adds direction embedding on the basis of position embedding and tag embedding, and decodes in original and reverse character order at the same time. The model generates two predictions for a given sample, and uses cross entropy loss to calculate the probability of each type of true value. Finally, the output with the highest word probability will be selected as the final prediction; (3) Obtain the final prediction result.
2. According to the Transformer-based bidirectional scene text recognition method of claim 1, it is characterized in that: Step (1) is as follows: for the training data set, a synthetic text data set is selected to train the network; the network performance is evaluated on seven common test sets, including the handwritten text data set IIIT, the scene text data IC13, the scene text data C03, the scene text data IC15, the street view text data set SVT, the street view text-perspective data set SVT-P, and the curved text data set CUTE.
3. The method for bidirectional scene text recognition based on Transformer according to claim 1, characterized in that: The same decoder network is used to condition the reading direction of the output sequence by adding a directional embedding. In each training iteration, each input image in the batch is decoded twice, from left to right and from right to left.
4. The method for bidirectional scene text recognition based on Transformer according to claim 1, characterized in that: Unbind the position encoding strategy in the input token embedding, calculate the position attention, and then add the position attention in the multi-head criss-cross attention mechanism; The position attention formula is as follows: ; ; in, j Truth types representing different directions, represents the position embedding of the encoder part, represents the position code, , , , is a linear layer, d represents the dimension of the linear layer, and all layers in the decoder share the same position attention; and Both indicate position attention.
